I believe I found a pattern to their behavior. I noticed this with ships and found it also applies with land units. The AI is set up to recognize the difference between pathed units, and units moving sporadically (i.e. different path every turn). If the AI has a unit set on a path that is meant to move through your territory but not land on it, then it will give you the move NOW option because it expects it should technically be out of your territory next turn even if it isnt.
If a unit is trapped, cant find a way through your territory, or is blocked, and you already gave the Move NOW option at least once, you can then do the declare war option.
I came to this theory when I was in the industrial age and pumped out lots of privateers. I found that as long as the AI entered my territory strictly to attack privateers, not meaning to enter my seas (it just happened to be that the privateers were in my territory), he could stay there as long as he wants. But if they all die (or I trapped him by sending 8 ships to surround his one) then I get the declare war or move option.
Afterwards, in the modern age, I found that if he moved destroyers across my tiles in order to cross seas and not meaning to stop in my territory, then you only get the Move NOW option. It does this with marines and tanks as well (the AI sent his troops through my territory in a few games Ive played) if they are passing through to attack someone on the other side. If they are exploring, or pathed to end on a tile in your control, then you can declare war.
I might be wrong. But this theory hasnt failed me yet.
