The 'exploit' I previously posted (about setting captured annexed cities to raze status temporarily to get a policy or tech earlier) only applies to my modded DLL where costs are based on the current city count - it hasn't been around for 8 years
After further examination of the code and some debugging, I believe that the case where the max count gets increased for liberated or immediately-razed cities is a bug, or at least not working as expected. It's also not working like the vanilla game without a good reason that I know of.
The code G posted includes this comment:
// Don't count cities where the player hasn't decided yet what to do with them...
It then tests IsIgnoreCityForHappiness() looking for such cities. This examines a flag set during the popup where you choose what to do with the city.
I have seen that the routine GetMaxEffectiveCities is called (many times) on conquest
before the popup appears. On the first of these, the city is not ignored and the max count is increased. Later, it is called during the popup and would then result in the correct number of cities, but the max count has already been increased.
As an aside, I think that the case mentioned by chicorbeef, where when you save a game before exiting the popup you are then not penalized for the city, is probably caused by the flag being permanently set. I'm surprised that G doesn't consider this to be a bug.
I think I have also found a workaround/exploit/whatever for the liberate/raze situation, provided you have previously captured at least one non-capital city. If you set that city to be razed, then capture and liberate or raze another city, then unraze the original city, I think the max count won't be increased. I'm not well placed to test this now, so I would be grateful if anyone could confirm this.
If the liberate/raze case could be fixed, it would still leave the question of whether changing to using the current city count to determine penalties would improve the game from the point of view of a civ that loses one or more cites being able to recover better, both for human and AI players. However, I suspect that fixing the liberate/raze case will be difficult without introducing new problems. Changing to penalties based on the current city count would avoid the need to fix that case, probably avoid the save-during-popup issue, be easier to explain and understand and possibly improve the game. It would, however, need something to prevent to the 'setting captured annexed cities to raze status temporarily' exploit, probably making cities being razed count until they are gone.
Edit: From other current threads, I see that puppeted cities can also increase the count in the same way as liberated or instantly razed cities. I very rarely puppet myself and hadn't seen it mentioned before so I wasn't aware.