ok, many interesting comments here.
trying to condense the points:
0. non-expansion
1. settling aggressively in indefensible locations
2. useless cities in bad locations
comments/questions in response:
0. seen that a couple of times. they don't build settlers. root cause might be as simple as AI not researching pottery in time, which can happen through bad luck, because of random choices. gazebo topic, maybe solved already.
1. many aspects to consider here:
* what is aggressive? some AI players are supposed to be aggressive ... i assume it would be a good move if they put enough units there to be able to defend the city? so in fact it's not really about the settling but about the defense, right? what it boils down to is that the AI needs to be able to judge the likelihood of an attack. and that's very difficult to code. who is a pushover and who isn't? who has room to expand and who is desperate?
* anyway, there's already a check for military power, if the neighbor is stronger they are supposed to keep their distance. there was an interesting proposal "don't settle near developed cities of powerful neighbors" ... sounds good but how to define developed? also this would lead to areas of no-mans land between players. the more powerful player would be able to grab it without a fight. is that an improvement?
I suppose you could use the economic power of the city (used for warmonger penalty when capture), but I don't know if it is doable, or anything which is the approximation of the age of the city.
The undelying idea is that forward settle a wide empire and forward settle a tall empire is not the same thing.
In truth, the main problem is not that they settle those cities, the problem is that they don't expect the human to declare war to raze them, so are not prepared to defend them. Maybe they underestimate the importance of the diplomatic penalty for forward settling ?
Another idea: is it possible to add a "connection check" ? which make sure that the AI is able to secure a reasonnably short path from its army to the city. The worst case for defense is when the city is isolated from the main empire.
For no mans lands, If the city is settled at ~6 tiles of distance from the other civ, there is not any remaining land, and not that much contested borders compared to settling at minimal distance. Moreover it give more space for defensive fight. Also, I do not advocate strict rules, forward settling a natural wonder is worth it.
2. again, many aspects:
* what is a bad location? a city has 37 plots in its three rings. with specialists and building bonuses etc it's very rare that all plots are being worked. so some overlap between cities is acceptable. but of course that's a matter of personal preference and the AI leaders have differences here too. in my experience, even marginal cities will grow (especially with a food trade route) and become productive eventually, due to the bonuses from buildings. the terrain is surprisingly unimportant.
* so how many "good plots" does a city need? what is a "good plot" anyway?
* a human will sometimes turn a good site into a bad site by rampant plot buying. i might discount the value of plots adjacent to enemy territory, but the real lesson for me is to add a cooldown between plot acquisitions.
How many good plots : well, usually, the reasonning is the converse. I want a city in a particular area, for luxuries, for defensive purpose, for land grabing, ...
But the AI is usually fine with it. The only problems is when there is no longer any "good enough" spot, and the AI still has settler to use. Then they either wanders, either settle crappy forward settled cities, while they should probably either wait for the upgrade of the settler, either settle a city in a more conservative way nearby their territory, taking advantage of the tiles no longer worked by develloped cities working specialist instead.
For plot buying, I don't think puting a cooldown will really solve the problem. It would balance it better, but would be very frustrating for the player. If any cooldown, I would forbid to buy tiles the turn you settle a city, in order to make sure the forward settled civ is able to buy the tile "it should have".
Personnaly, when settling, I consider that tiles at 3 tiles or less of a city belong to the city (or only 2 tiles for a "newborn" city), so I don't count them (except if I have the gold to purchase them immediatly, but I feel like cheating when doing so).