First off, thank you to 45* and Rezca. I picked Rezca as a benchmark to compare the AI against in some fine-grained debugging. In the first 60 turns, Rezca's save shows he has 3 cities and a score of 76. This is pretty good.
I ran a game with a debug DLL and let the AI play for itself for 60 turns. The initial results were not so great. The AI had only managed to found 2 cities, (a pitiful size 4 capital and a pathetic size 1 city) and had a meager score of 52. Rezca outmatched the AI by nearly 50%.
Save 1, Poor AI Run (60 turns in)
I began taking a look at the AI's actions in the Better AI log. Despite having only 2 cities (one of which was brand new) the AI had 7 military units. It was wasting a lot of production on military, when no one was even close by (only 2 others met, both far away).
I slowly walked through the code that determines how many defenders the AI needs. The code starts out with the assumption the AI needs 1 defender per city. Then an addition one for capitals. An additional defender for recently founded/captured cities. And on and on.
The result was the AI thought its capital needed FOUR defenders. Maybe later in the game this will be true. But at turn 60, no way. I added some logic to shortcut the normal defender additions for capitals & recently acquired cities in the early game. I hoped this would be enough. I recompiled and let the AI play from the start again.
Failure. The AI did not even expand past its capital. It kept trying to churn out wonder after wonder. It had a settler built, but was not moving it. It had built the settler about 20 turns ago, and it had never left, despite 3 military units in the capital. SO... the problem wasn't just wasteful military production. Why was the AI not settling with its settler? I slowly executed a turn at a time, walking through the AI_settle() code in the UnitAI. The AI settler puts out a contract for an escort, which is fulfilled. Then the AI settler hits a part of the code that tells it to retreat to the nearest city...What? The AI was breaking up its escort each turn. Only in rare cases (very close by, good city spots, or city locations on another continent/island) would this retreat check fail. I removed it. I recompiled. I loaded up the save. Next turn, the settler took off, heading to found a city. Problem solved. (Note: I did not make a save of this run, due to me advancing past 60 turns to debug the settler issue).
Take 3: I started the initial save again, with my last 2 changes. The AI built a settler after building an initial javileneer. It had a 2nd city in the first 20 turns. It grabbed Stonehenge. Then some more military. Then a third settler. Before 60 turns were up, the AI had 3 cities, a score of 78 (A full 2 points higher than Rezca!) and combined empire population of 11. Things are looking good so far.
Save 3, Good Run (60 turns in)
Things look better, at least early on. But could this game be a fluke? A few good rolls of the dice? I reloaded the start save and ran the AI again. From the start, in 60 turns the AI had again 3 cities, a combined population of 13, and a score of 86. Even better than the last run, and a full 10 points better than Rezca. So my changes are not a fluke. Early AI: A success!
Save 4, Good Run (60 turns in)
I'll be pushing the AI enhancements later this evening, and using Rezca's 60 turn save to run the next 60-120 turn batch of save games.