We all know (and agree on) that the AI is programmed to go for all unsettled places (especially, when there is a resource there) and to produce settlers as long as there are still unsettled spots. So could the incidents that you have observed many times (namely you sending a settler to some spot and seeing an AI settler arriving there around the same time), simply be a consequence of that algorithm? Or in other words, the AI is programmed to spam settlers and send them to still available places. It does so all the time and it does it whether or not you are also trying to settle that place around the same time. You (or the human player in general) are/is also trying to grab space and therefore it is only natural that on some of these occasions you two are going for the same place and the AI beats you to that place. But that doesn't mean the AI did so, because it can somehow predict your intentions and is trying to thwart them -- it did so, because it does what it is doing all the time: trying to settle open spots within reach!
The above would also be a plausible explanation for the behavior you have observed, and a much simpler one than the explanation of the AI having highly advanced human behavior analysis algorithms built-in so that it can predict our intentions and counter-act them just in time.
And by Occam's Razor, if there are multiple possible explanations for something, we should prefer the simpler one (meaning the one that requires less assumptions). Unless proven otherwise, of course. But the only 100% proof in this case here would be to look at the source code and see, whether these algorithms are really there or not. And as long as that did not happen, you have to accept that others have a different opinion on that point than you.
We're talking about a computer here, and, as you yourself point out, a computer doesn't tend to 'act randomly'. It functions by code.
Yes, the easy answer is that it's 'coincidence', however, by playing the game regularly, one notices that there's more to it. Factually, not by opinion. Can this still be 'coincidence'?
How can coincidence explain this:
From a previous screenshot I continued my game. I scuttled all the way to Steam Power before backtracking to Navigation. The first thing I see once I'd done just the most cursory exploration is this group of tasty looking islands just to my east and just to the Inca's north:
As you can see, a whole chain of lovely islands, all of which have resources on them, there's also horses, Iron the whole caboodle stewn around the long island complex.
You'll also notice that the Incas are well within easy reaching distance of them. At this point in time there's no logical reason why the Incas haven't settled them yet.
So I check to see whether the Incas have even learned Map Making yet or not:
As you can clearly see, they are well past Map making, by quite a huge margin. And their towns and land do not appear to preclude the building of Settlers nor ships.
If the AI ALWAYS 'simply fills the space', why hasn't it filled this space?
So I set about pumping Settlers from a couple of Settler farms and prepare to settle these lands, as they seem quite nice.
The exact moment I launch my ships full of Settlers, this happens:
As if by magic, the Incas are now suddenly stirred into activity.
You will also notice that the Incas are on a secluded island, and so they have not been too busy having wars, in fact I'm the only people they know exist (trading & etc-wise), and they have a full island that's not very big and have not been too busy Settling other lands.
Since we last spoke to them they have learned Feudalism, so their tech isn't especially shabby and I think it's safe to assume they got Map Making long before they passed into the Medieval Age.
I honestly can't believe this AI chose, uniquely, to leave Map Making until the very last tech it learned in the Ancient Age, I have never seen that before.
And, next turn, here's my first of two ships full of Settlers:
And you'll see the Incas are now, all of a sudden, furiously settling these lands that they haven't been bothered about for hundreds of years, maybe even a thousand years.
Now, as you say, a computer is just operating by code.
All the other AIs in this game have settled their reachable lands, some occupy many little islands.
It was just these guys who decided not to because...? Because because because...
Now, I think my theory works. Yes, my theory is still an opinion because we don't know why such things as the above happen, but my theory is the Razor theory here. You might well have an opinion that it's all coincidental and explainable by existing knowledge, however, it is your version of events which, the more one looks and takes note, appear to be more in the conspiracy side of any debates about Razors.
What kind of 'evidence' do you require to 'suspect' there's something 'code-based' going on if the above three examples are not, by themselves, completely definitive?