And here was I thinking that we'd closed this case already ...
Okay, here we go:
1. Is a realistic, contingent scale hidden in the game's data?
I don't believe that the scale (which you claim to be present in Civ4) actually exists in this form. You claim that the "soldiers" factor used for different unit types, as they are used under the hood for the calculations in the demographics screen, are actually hidden "hints of the developers". The problem is that the numbers just don't hold up.
In this hidden scale, a caravel is worth 3,000 soldiers in the demographics screen. Now caravels during the age of discovery usually had a complement of about 20 men, so by your assumption that these metrics make sense, 1 caravel unit built in the game would represent about 150 caravels in the real world. This means that when I send a caravel unit out to look for a new continent, I'm "really" sending a fleet of 150 ships. That's a bit odd considering that Columbus only needed two caravels and a single carrack.
A submarine, according to this hidden scale, is worth 30,000 soldiers. The most popular German U-boat in WW2, the Type VII, had a complement of about 50 men, so let's say that one submarine unit in Civ4 equals about 600 vessels. That's already pretty close to the
total number of Type VIIs involved in the complete war (about 770), and it's about half of the number of
all German submarines of all types involved in the war. Again, this is a single unit in Civ.
An ICBM, according to the scale, is worth 40,000 soldiers. Now it becomes
really tricky because, well, where are 40,000 soldiers near an ICBM? Missile base personnel? No, actually these 40,000 men must be crammed somewhere
inside the missile, because when I detonate it, they
go away.
I might add that for the same calculations in the demographics screen, each factory is worth 2,000 soldiers, each levee is worth 2,000 soldiers as well, and Cristo Redentor somehow has 10,000 soldiers hidden behind his back (as have the Moai Statues).
Given these numbers, I think it's a bit of a stretch to claim that these represent a contingent scale, hidden in the code by the developers. Even if they may have done some research in that area (I wouldn't totally exclude it), I'd say that associating 40,000 soldiers with an ICBM and 10,000 with Cristo Redentor is a clear indication that at least some other factors contributed to these numbers as well, and there's a good chance that there wasn't even a truly "realistic" scale to begin with.
I don't see a good basis for the second premise of your theory either, i.e. that the game is "clearly meant to be played on an Earth-sized map". Given that the game can be played on a variety of map sizes I don't see how such a case can be made.
I'm sorry, but I just don't see that the contingent scale that you try to preserve is actually there.
2.
Should there be such a scale?
That's a different question. Arguably, even if such a scale isn't really present in Civ4, it might be worthwhile to create one and/or make sure that any proposed additions and rules changes adhere to one. However, I have to admit that I don't really see the merit here. I don't think that it's feasible, or even worthwhile, for a game on such a grand scale to try to mimick Earth history to such minute degrees. I don't see what it would add to the enjoyment of playing it, while I
can see how stack counters could add to the enjoyment by increasing the strategical depth of the fame. However, I do realize that if such a scale is important for you, then the suggestion I made may not be very attractive for you. That's fine as far as I'm concerned, as different people obviously want different things from the game.
In that vein, for what
I want from the game, the whole scale you propose and the whole discussion in the first section of this post is actually rather irrelevant. I was checking out ways to increase the tactical options in the game by adding counters for a game mechanic that (apparently) many people see as overpowered. I clearly came from a "game design" angle, not from a "real history simulation" one. Now if someone asks me what a real-world representation of a proposed game mechanic might be, I can come up with some, and I listed some which, in my opinion, did the job well enough. However, the question itself still isn't very relevant to me. If I wanted a totally and unquestionably accurate simulation of historical warfare, then I wouldn't play a game that sends catapults on kamikaze runs against fortified defenders, would I?
3. Other ways to limit stacks
As I said in my previous posts, I think that there are many possible ways to limit the power of stacks - I mentioned some myself, you mentioned two more (thanks for that). Personally, I don't think that limiting the number of units per stack to a hard cap is going to improve the situation, because in all games that employed such a mechanic, I've seen players build "killer combos" of stacks which had all available slots filled and which were then nearly unbeatable for the AI (because the AI, with all its production bonuses, leans heavily toward a "strength by numbers" approach, which is crippled if it's impossible to field a larger stack than the player). The problem I see with "stack congestion" is that it's a mechanic that punishes an otherwise optimal behavior rather than countering it. In game design, this is usually not an optimal solution, since there's a good chance that players will feel restricted rather than challenged (which in turn leads to less enjoyment). On the other hand, for a mod that people choose voluntarily, that might not be that much of a factor.
Anyway, I'm not claiming that I know the only solution. Actually, as I said, I believe that there are many ways to limit the power of stacks if one wants to. Obviously, some will meet (or fail to meet) the preferences of different groups of players. It's become pretty obvious that the counter-mechanic suggestion I made doesn't meet yours because it collides with your impression of scale in Civ. That's okay. I wouldn't call a suggestion "poor" though, if it accomplishes what it was designed for (which I'm fairly sure my suggestion would do) but just doesn't cater to my vision of the game and how it should be.