Not sure if this was mentioned already, but I think proximity to a city makes a difference as well, guess that goes along with ho much of a threat the unit is to their empire. I remember reading somewhere about how the game determines if you ask them to leave, or tell them to leave or declare war, just can't remember if it was in a thread or an article somewhere, jsut remember that if a unit is close to a city, you're more likely to ask them to leave or edlcare war.
Many times to connect trade routs, I send workers into Opponents territory, usually they ask me once to leave, which I agree to, but still keep my worker, working away on a raod, with the AI seemingly not bothered by the worker anymore.