Kael did something with the beeline issue in 0.34, hopefully got rid of it rather than just placed a check for if the tech is allowed or not for the game. Depending on what he has done we'll see if further tweaks are needed on that line.
I'll be playing with the way that cities decide what to work on when I get into the code again (possibly this weekend). I intend to make the AI work on buildings which enable units for them as a priority, then work on defenders (and increase the desired number of defenders in general), then it will probably stick to the old priority system while I see how that works out.
All in all, the basic rundown of any AI function is a simple series of checks done in a specific order. Whichever one is set first in the order will be done the most. I believe that what we are seeing is that in BtS few units actually REQUIRE a building, so the first check done is to see if there is a need for any units, and if there is to fill them with the best it can currently make. In FfH it is suicide to keep working on outdated units, tossing them to their deaths, and building replacements. You need to get out your vital buildings, THEN ensure sufficient units.
After some work on prioritizing proper build orders in cities, I'll start looking for all uses of AIWeight and removing them. They are nice when you are trying to inform the AI that there is a nifty Python function that is good (like Planar Gates), but it is HORRIFIC when the AI is trying to make longterm plans. If I am seeking to get the AI to build good defenders, I will have it prioritize defensive strength, and free promotions that enhance defensive capabilities. If I am seeking to get the AI to build good offense, I will have it prioritize Collateral Damage and Offensive strength. But in both cases, AIWeight is JUST AIWeight. I cannot tell the computer if that means "Defend Good!" or "Generate Money!" or anything else. Overall, it is something which should have only been used as a band-aid, but has become a crutch.
About the same can be said of Flavors I suspect. They are meant to be small, just to provide some "random adjustments" between each leader. But instead they are currently fairly large, and thus they dominate over everything else. Thus instead of considering 2 technologies as "This one enables a Wonder, but the other one enables better tile Yields. My economy is floundering, so I cannot afford to work on a Wonder, so I'll take option 2" the AI is forced to say "This one is Culture, which I LIKE, and this other one is Science, which I don't like. So I'll take the first one!" In the appropriate case, it accounts primarily for the current situation, and uses the flavor of the Civ/leader as a balancing factor for close tie situations. In the inept case, it just seeks out arbitrary flavor which may not even be properly set up.
Anyway, all of that is just so those of you who don't think "computer-esque" can understand that it can be quite tricky deciding how to go about making the AI "smart" about things.