Castles (plural) are rarely of any real use unless you need them for defence. A castle, on the other hand, if placed in your most powerful trade city (especially with ToA), can be very nice indeed.
The key to making loads of
from trade routes is to concentrate the best routes in the best cities; but anything that gives more routes to your top city will take those routes away from lesser cities. If all of the best routes are going to your capital/ToA city, then extra routes elsewhere aren't going to be worth that much.
The castle gives you so little benefit other than the trade route (and that doesn't last), that it's usually not worth the effort to build it anywhere except the main trade-hub.
The key to making loads of

The castle gives you so little benefit other than the trade route (and that doesn't last), that it's usually not worth the effort to build it anywhere except the main trade-hub.