Even in civilizations that were baseline maritime, like ancient Greece and Phoenicia, not all cities were coastal. Sparta is as far from the coast as it can get.
Babylon is not coastal, Egyptian capitals were not coastal, Rome is not on a coast, civilizations in India were not coastal, none of the native american civilizations were coastal.
I can turn this argument on its head and it makes just as much sense: Russia is not really a land power because some of its important cities are on the coast: Saint Petersburg, Vladivostok, Murmansk.
Oxford is not on the coast, therefore the sea was of no importance to England?
And Rome was not on the coast, but Rome got its food from the coast: Ostia at the mouth of the Tiber was one of the busiest sorts in the whole Empire, and grain shipped in from North Africa/Egypt fed Imperial Rome. When the Goths cut that trade, Rome's population fell precipitously, because no land route could provide the same amount of food.
You could say exactly the same thing about Athens - which also is some ways inland from its port of Piraeus, but got the bulk of its food, grain, shipped in from Greek colonies on the northern shores of the Black Sea. Again, when Sparta won control of those trade routes with her new Persian-financed fleet, the Peloponesian War ended abruptly with Athens' defeat.
Both cities owed their existence as major metropolises to their access to the trade from the sea. The alternative, as your examples of Babylon , Egypt and the Indus cities indicate, was access to major river transport - not for access to fresh water, but for transport of food in bulk.
The game currently and correctly allows a city not on the coast (like Athens, Rome, London, Antwerp) to still build a harbor and access the Sea for trade. Unfortunately, as this and at least two other Threads are discussing, it severely penalizes cities that are actually on the coast, like Saint Petersburg, Alexandria, Byzantium/Istanbul, Shanghai, New York City, and that simply makes no sense in either historical or game terms.