I can't blame the AI for forward settling as it was something I got used to do as well in Civ V (claim the contested good spots, then fill the gaps).
However, AI behaviour in Civ IV is specially irritating because of two things:
- " In your face" settling, directly competing for 2-nd / 3-rd ring Hexes.
- "In your back" settling, which means he is placing a city behind the ones you have placed (or even your capitol), without possibility of giving continuity to his empire.
- Plus the non-knowledge of what resources will appear, make some placement decisions really weird (bad land).
Imho, these are things to be addressed: AI should not compute -at least not with the same priority- resources they have no have access to or have no immediate use, AI should take into account the annoyance factor of border fighting (it's not the same spacing your cities 4-hex than spacing 4-hex with a neighbour), and AI should consider the posibility of connecting to his empire (connecting is posible if the land next to nearest border is empty. It is not possible if you have already an enemy city in the middle - unless you plan to declare war, that is, but then you are granting your neighbour an easy target as well).
Tangentially, taking into account loyalty will have the effect of considering these two points (near enemy cities provide more negative loyalty pressure, as well as being surronded will do). However, they should take into account the resource thing to make the AI appear more sane.