When your territory EXPANDS, animals won't automatically LEAVE it. Which makes it annoying to have hunters CHECK the new tiles each time, lest I lose workers to animals.
If you mean that they ENTER your territory, though... I'm yet to ever SEE that happen, lol.