I think maybe we could take a page from Mideval: total war, concerning the castles. Castled cities are entirelly different from castles. Pretty much every large town in mideval times (as well as in warhammer) had a castle, but only the greatest Keeps actually contained the entire city within its walls. Most cities had a castle which was surrounded by an unwalled sprawl. Therefor. although i do think that you idea for castles is overpowered, you could make it so that if you want to build one you incure either :a moderate to severe economic penalty in the city or a population cap. Even the largest castles of the middle ages could only hold a moderatly sized city within thier walls.