The Tower of London, Warwick Castle, Nottingham Castle, Hastings Castle and dozens of others were built immediately adjoining the towns they protected. It kind of defeats the purpose of having a castle if it's too far away for you to retreat there.
Unfortunately for your point, none of those castles were built to 'protect' the towns around them. They all started as Norman castles or Motte and Bailey forts built in 1066-67 to protect the Normans from their (rebellious) Saxon subjects, and keep down uprisings among those subjects. Rather a Reversal of the 'classic' Castle Function.
Since the majority of towns in continental Europe already had walls/fortifications protecting them by the 10th century CE, a castle in the town was redundant for the town's protection. On the other hand, when facing fast-moving raiders (Vikings, Magyars) you needed a fortified place that the villagers living Away from the walled town/city could run for when the alarm was sounded. You also, ideally, needed well-equipped troops fast enough to catch the raiders, which is where the mounted, armored knight in the castle provided both protection and Retaliation all in one.
I think that we need two mechanisms here:
City Fortifications/Walls, which we already have but need to be extended to include the possibility (with lots of Gold) to protect all or most of the city Districts as well as merely the city center. If you look at preserved medieval towns in Europe (Rothenburg-ob-Tauber is a perfect example) the city walls had to be expanded several times as the town grew, resulting in the photo-ready interior gates of the old, demilitarized walls (they provided solid back walls for new buildings and masonry 'fire breaks' to stop fires from spreading) throughout the town. Eventually, in the Industrial Era historically, the cities grew so large you had to put detached Forts around them at a considerable distance from the city center, and those subterranean, concrete/steel, artillery-filled forts were so hideously expensive that only a very few cities had them.
Conclusion: If you have the resources, allow the 'city wall' to be extended around all the districts touching the city center directly or through another district. After the initial 'City Center' wall costs could be calculated by the number of Tile Sides of Wall required. Add a District, and, as historically happened, you either plunge for more Wall or tell the people living there they are on their own - and every walled city since at least ancient Athens had People Living Around (outside the walls).
Forts protecting something important in the countryside, away from the cities and districts. These are buildable from the Ancient Era with, say, the Tech: Masonry (which also unlocks the cities' Ancient Walls) and, possibly, also the Civic Military Tradition, since they were frequently built by a local 'noble' or 'Big Man' to protect 'His' people and local power. Or, they are buildable with Masonry, but with the adding of the Civic their Maintenance Cost drops to Zero and they receive an Intrinsic Defense/Hit Points equal to 1/2 a city center with Ancient Walls.
With the Medieval Tech: Castles the Forts can be Upgraded or newly built as Castles, which with the Civic Fuedalism provides the 'benefits' of the Feudal Knightly Class: The Castle takes points of Gold/Production/Food out of the Tile, but not only has a (greatly) increased Intrinsic Defense/Hit Points, on Declaration of War provides a Knight (Feudal Retainer) free of maintenance.
When Bombards become available at Tech: Metal Casting, you can start converting Castles into Manor Houses, Chateaus, Rezidenz, or any of the other terms used to describe Stately Piles of Stone Full Of Dusty Old Aristocrats: which give Culture boosts and might even provide a 'slot' for a Great Work of Art, and later on will provide a Tourism Boost as well. And, if an enemy at some time has already destroyed the Castle, it becomes a Ruin (nice additional graphic on the map) which, in turn, provides a Tourism Boost in the Industrial and later Eras for the Tile.
