I've also noticed that military units on the same tile as workers building improvements have a way of muting the sound of their work.
Whether they're hammering, raking, digging, or whatever, that military unit being on the same tile as your workers makes it impossible for you to hear what's going on (you'll only hear the military unit). This could be a way of making it just a tad harder for you to snoop around and learn what a player is up to.
Sound CAN carry over several tiles, and terrain features (forests, hills, and --worst, because they echo--mountains) can affect this. Try zooming in as far as you can, and listening to what you hear when you're focused on a particular tile, and you'll see what I mean. Turn the gird on and move to adjacent tiles and notice where and when you CAN'T hear certain sounds.
There's more than meets the eye (and ear) here...