There are a couple reasons you might build a castle (For the sake of this discussion I am going to assume the wall is included in the cost).
At Normal speed, castle is 150

, 75

with Pro xor stone, 50

with Pro AND stone.
Only under exceptional circumstances would I suggest building a castle with neither Pro nor stone. Otherwise, the building is often worth considering.
Firstly, if you have ToA or a city that is earning typically at least 2

from each trade route, the trade route may be worth a bit more than you let on.
Secondly, for 50

(Pro and Stone) you get a monument plus a trade route. Considering the Cothon costs 20

more than the Harbour, this seems to fit exactly if we assumed a trade route was worth 20

(I know thad sounds weird but never mind). Also, you can time the whip of the wall so that you can do a roughly 1pop whip earlier than you'd normally be able to do for a 50

building (I think that's right??)
So with Pro and Stone, you could easily argue castles are better than monuments,
particularly in MP where that 100% defense in a new city is awesomely intimidating, and the trade route helps get that city profitable asap. The library gives the 2

for 90

and is arguably the better way to get culture instead of using monuments. If Creative the library would win hands down obviously.
The thing with the castle is the separate bonuses always look weak in isolation. Yes, the 1 trade route is weak, yes the 1

is weak, as is the fact the defense can be taken down with a spy and the 25%

doesn't seem worth it. Together though they make it at least average IMO. If one were to employ binary espionage, meaning every now and then you blast 100% espionage from your commerce sliders, that 25% could be quite nifty.
Overall, I would argue the castle is mainly for Pro leaders, just as the Drill line of promotions is primarily for Pro leaders.
If the AI were half decent, castles would be one of the best buildings in the game if you had one of the multipliers. In MP, stone or Pro-boosted castles would be much much better than in SP.
EDIT... Actually the whipping of walls is extremely risky now because of the bug in 3.19. Whipping a +200% multiplied wall will waste hammers a lot of the time.