Some warmonger policies give bonuses to walls and castles, too.
Even if you don't want to fight a lot, you may need some walls for defence against barbarians, and it definitely helps with crime unhappiness, as it increases city defence slightly (garrisons are usually better, but it adds up).
Check the happiness guide in the strategy section.