Well, you can ask them to leave immediately as well if you want. You can't make them leave unless they stay too long. However, the same applies to you - you don't have to leave just because they asked you to.
I often build roads, irrigate, etc in AI territory.