The problem is that the current system was developed with vanilla in mind, in particular the later eras where urban fighting in large cities (think Stalingrad) provided far more resistance to an invading army than any city wall ever would.
FFH is based in a high-fantasy setting though, and high walls should be far more telling than the defensible positions provided within a large city centre. At the moment, you get the counterintuitive result that a monolithic fortress in a remote mountainous area is more easily taken than a village populated by artists and poets.
The solution would be to drop cultural bonuses significantly (down to 5-10% per level maybe?), and drastically up the defence bonuses offered by walls etc. to take their place, maybe adding a bias into the AI to encourage them to build them.
