This is perfectly normal.
What is going on is that the game engine "caches" (basically stores & remembers) lots of values, tile yields, building effects, etc. When you build a new building, for example, it add the new building effects (from the XML values) to the internal cached values. This saves a lot of time, because otherwise the game would have to check and add up the values for each building the city has, each time it wants to know a value.
Occasionally these values do not get updated correctly. This is rare, but there tiny bugs in processing (or even division/rounding-errors) can result in the accumulation of incorrect values over hundreds or thousands of turns. I wish all of these bugs could be caught and fixed, but they are incredibly hard to spot.
What recalculating the modifiers does is clear the internal caches. The game recalculates all of the internal values from scratch. This gives you a "fresh" state, and is why your gold or yields might change a bit.