no, actually the default pathfinding will put a unit on a defensible tile all the time over open terrain. it avoids open terrain when possible at all times
however it will not for example calculate a "detour" where it reaches x,y using the best defensible path, it searches for shortest path with defensible tiles. its not the best, but it's the quickest, which is really important in the early turns for the AI to found a city: every turn counts
As far as the AI is concerned, the open terrain penalty *should* make open terrain literally a no-go for it.
As ohioastronomy correctly points out below, due to the traffic jam problems of 1upt the AI nevertheless is "forced" (by bad code) to move units into the open anyway.
Because this "forciing" is a result of other units already having moved the AI is incapable of planning for this which ultimately leads to the fact that AI units in open terrain typically are easy prey for the human.
Not when there are too many units, and not when terrain is constricting. There is also balancing the fog of war - e.g. not knowing when it is safe or not because of enemy units just outside of range. The too many units bit is important because that is the main equalizer that the AI gets in this series (more units), and the architecture neuters that.
but overall though, at its basic level, firaxis made the pathfinding chose defense over open. if only they tried harder to consider the kind of conditions you listed, might just of made the AI that much more smarter
But with the open terrain penaly the weaknesses of AI pathfinding (together with bad decisions about the sequence in which units move) become even more obvious.
Apart from this an open terrain penalty is across all historical experience.
Battles throughout almost the entire history were fought in open space. Part of the medevial combat doctrines was that the attacker would be at a disadvantage as he would have to expose his troops to enemy archery fire.
The Civ5 combat system with its various sub-systems completely turns early time warfare in its opposite. There isn't any plausibility in the way in which we are forced to use troops.
Worse, the open terrain penalty even punishes ideas like flanking the enemy by making use of higher movement rates in open terrain.
In total, the Civ5 combat system is much more a perversion of real life experiences than Civ4's combat system could ever have been.
By nerfing down the penalty the devs admit to see this problem, but simultenously show to be stubborn enough to avoid to make the necessary changes as long as ever possible.
One could say that the poor state of AI warfare as of release was due to a rushed release. That they are not willing to make basic changes, yet only do some cosmetics, two months later puts them even into a worse light.
