Yeah, that's fair. I think it depends on the overall design of the game. For a game like Age of Wonders, the tactical battles are the main course, and the 4X map stuff is there mostly as a frame to support combat. The battles there are often long, and the tactical combat system is complicated. For a Civ-style game, the gameplay is mostly on the world map, with the occasional trip into some menus. I can see how an involved tactical combat system could take you right out of it.
Fallen Enchantress is interesting to me, because it falls somewhere in between, and I really liked the balance. There's a solid 4X component, and tactical battles are of medium complexity and length. Perhaps the mix is more suitable for a fantasy 4X than a historical one spanning thousands of years though.