I echo Boris' sentiment that the magic is with the complexity of the game. Civ 6 is complex enough that there seems to be a never-ending list of experiments worth trying. The game's complexity is results from the game being a web of many many mini-games. Each mini-game is usually pretty simple, but what makes the game complex is the number of these mini-games and the different ways they relate to each other.
For example, scouting maybe an integral part of the 4x genre, but it's really a very simple (and frankly, boring) mini-game on its own. There's very little skill involved in sending units out to gain information about the map, precisely because there's so little information to make informed decisions with. Yes, there are skills involved in scouting, like utilizing the settler lens and paying attention to the edges of tiles that are hidden in the fog, but you don't need to play the game a ton to be able to use these skills well. For it to be interesting, the scouting mini-game HAS to be tied to other mini-games. For instance, there's a mini-game of deciding what to produce in your city. Should you open with a scout, slinger, warrior, builder or monument? I think the answer is usually very clearly a scout (two scouts, in fact), but plenty of people will disagree with me, and I acknowledge there are instances where it doesn't make sense to open with a scout. The connection between the what-do-I-make and scouting mini-games forces the player to think about the relationship between production and the expected value of what a scout can get (e.g. gold, builder, eurekas & inspirations, era score, map knowledge). Another mini-game of what-tiles-do-I-work forces you to relate the value of a scout to growth, culture, production, etc. Do you sacrifice 1 production (or even 2!) for 1 culture when you're building your first scout? This will allow you to unlock Code of Laws faster, which allows you to slot in Survey just in time for your first scout to level up after discovering a natural wonder that you know from your warrior's preliminary scouting is very close to your capital. But, does that mean 1 culture is always better than 1 production? Not when you consider the inspiration feature or the wonder-building mini-games (plural because some wonders have on-completion rewards that force you to think about how and when you'll complete them). Most of the mini-games are very simple, and you certainly wouldn't want to play any one of them individually. It's the way these mini-games are connected to each other that makes the game great.
To answer some of the questions in the OP:
Q: Should the 4x's happen simultaneously?
A: If you see each X as a mini-game, they have to overlap in many ways to be interesting.
Q: Is the series headed in the right direction?
A: I can't speak for older installments because I haven't played, but I think Civ 6 gets it mostly right. My biggest complaint is that the devs don't seem to understand that mini-games have "expiry dates". As the game grows in scale, some mini-games stop being fun, but they stick around until the game ends. For instance, I'd say the what-tile-do-I-work mini-game stops being interesting when your empire grows to 20+ population (or maybe less). It's insane how the tile assignment feature is still available near the end of the game when your empire has easily over 100 citizens. And then, there are mini-games involving rock bands and archaeologists that are completely inappropriate for the game's scale from the get-go. These appear near the end of the game, and considering how many things you need to do at that point of the game, the amount of micro-control they require is absurd.
Within this framework of looking at Civ as a collection of mini-games, I can summarize the principles that I believe are critical for success for a civ-like game:
- Mini-games are simple
- Mini-games are numerous
- Mini-games are closely coupled
- Mini-games are phased out when they're no longer appropriate
Q: Is 4x closer to board game than video game?
A: I don't know exactly what makes something like a board game, but I'm guessing discretization of space and time is an important feature. I don't think this is an essential feature for 4x, but I don't think I'll want to play a version of Civ that's in real time, because imagining what it'd be like just makes me tired. Sid Meier's talked about how the very first attempt at Civ was in real-time, but that game lacked the depth that Civ has because it's unreasonable to expect the player to think hard about what decisions they're about to make when they don't have much time to make a move.
Q: Is immersion important to 4x?
A: Yes, immersion is always important in a game, but what counts as an immersion breaker changes depending on how we look at 4x. If we look at 4x as a sub-genre of strategy, then I just want to see that there's good logic behind the moves that my competitors make. If we look at 4x as a cousin to city-building and tycoon genres, there must be a sense of accomplishment when I look at the empire I've built. When I first started playing Civ 6, I cared more about the latter. Now, I care very little for it and focus almost entirely on the strategic aspects of the game.