I don't think the economics favor the defender anywhere near as strongly as you suggest, as evidenced by attacking being a rather succesful strategy in Civ IV, even against strong players who can defend well.
Si vis pacem, para bellum - if you wish for peace, prepare for war. And also "attack is the best form of defence".
That's why no peaceful civilization is going to survive if it doesn't have its own army to defend its territory and attack other civs when the need arises. War is an important part of strategy and empire-building, and nobody can neglect that. Stablishing combat modifiers for defenders in every type of terrain (despite of desert terrain titles) on CivIV was surely done to prevent peaceful civilizations from being utterly defetead by a warmonger one, but History has proven that is war that makes empires, not flower seeding.
Come on, the fastest way to get power is by harassing others for the goods they possess themselves, not by producing them yourself. War is, and will always be, the fastest way to get power, as long as you can pay the social and economic costs for waging war.
In real life, USA never stablished itself as a global power by sticking to "peace and love" and purely economic decisions. It's a global power because it has a large industry which supplies half the world with things they produce alone, it invests on research and technological development, but it also invests heavily on the quality of its army. That's equally true for China, Russia, England, Macedonia, Rome, Babylon, Persia and any other empire which has risen across History.
Having a small army because you invested heavily on research and cultural development woudn't prevent anyone from conquering your sitting duck "highly cultural" cities, because they're an easy target and the benefits from waging war against a pacifist and poor-defended civilization obviously far surpass the costs of war itself. That's why a peaceful civilization must have a well-sized army, accordingly to the size of the territory it needs to defend.
Being rich but peaceful doesn't make you less vulnerable to thieves. Having high walls and bloody hounds do.
In particular, pillaging and camping out on key tiles can effectively negate large portions of a defensive advantage as units will be forced to come out and attack if you want to regain access to important economic resources/squares. For instance, if you effectively defend a city, an opportunistic thrust at a key strategic resource can actually turn the tables, by pillaging and camping a moderate stack. I find the most common error people make in Civ IV combat is being too aggressive at going for well defended targets like cities when its actually rather easy to get winning positions by opportunistically taking what the enemy gives you.
Isn't being able to lay waste to improvements a realistic way of waging war? It has been done by all nations all across History. When medieval sieges on a castle took place in the past, the attacking army would poison the water supplying the attacked fortress, put crops on fire and such to either get the defender to surrender, or to defeat the defender without having to enter a bloodbathing battle.
When the Russian army retreated from outer borders when German striked in WWII, they burnt to the ground most of the cities left behind, which prevented Germany from benefiting from the newly-acquired territories. Isn't that strategy?
If you were the president of a nation and an invader started to lay waste to your towns, economic buildings and such, entrenching itself in the territory it has occupied, would you not send your army to get them back to Hell, even if you suffered from strategical disadvantages by being the attacking army? Isn't it realistic?
So, I can't see people's point when they state that CivIV isn't realistic. It could have a tactical map for resolving battles (as Space Empires V has), but although it wasn't implemented (unfortunately), it doesn't make the game less realistic, not to the point of not being worth playing it.
Finally, warmonger players always get upper hands in every empire-building games, not just in CivIV. Sitting behind a wall won't prevent you from being bombarded, nor it will prevent others from bombarding you. Attacking them from behind, cutting their supply lines or breaking inside your attacker's territory with a handful of fast units, laying waste to the now poorly defended territory (now that his units stand inside your country) will prevent defeat and force the attacker to reconsider the need of the war he has just declared on you.
