I switched to AI nations in one savegame and tried to figure out what's wrong with them. To be honest, I didn't notice anything dramatic there, they all just were in quite weak position compared to me. That was on Emperor level around 0AD
They didn't have any advanced tech I didn't possess, and hadn't developed any "super" cities. Their units were quite weak, with the strongest being light swordsman, defenders were archers or town watches. Two AI nations were weakened by a revolution, and Gandhi was totally owned by barbarians due to having very weak military. On the whole, having weak military was common, and for some reason their teching rate was quite poor. I really would like to take the time and really debug my saves to pieces, but work and family disturb my Civ4 gaming quite a bit.
One specific thing I noticed that almost all AIs had built the "Elder Council" which isn't that good thing to build in all cities as it obsoletes quite fast. It's very cheap and good to build in early cities though so it shouldn't be that much of a distraction for the AI build queues. Only a few wonders built by the AI in this game too.
As for using the recommended options, I'm doing that and taking away Education and Sports buildings too from distracting the AI from it's true goal in Beta4 (crushing me, that is!).
PS. One "annoying" thing, what's up with the Morale promo on the Confuscianism Shrine? Isn't it a bit overpowered? I go for Code of Laws as my first religion quite often, so I've abused that OP'dness quite a bit. Some bit less extreme promo might fit there better?
PPS. Unrelated: how on earth do I close the console window??? My son keeps popping it on by pressing the keyboard on random