Basically, if you want to maximize the number of archeological sites you get access to, never trade open borders to another player when you're closing in on archeology tech. If no-one has access to your borders, you will be guaranteed any site within own borders and can leave that one 'til others are cleared.
Secondly, try to grab any archeological site in no-man's land and in city state areas first. Prioritize those that are immediately adjacent to another civs borders (less he expands and claims the site) and obviously try to clear close to who-ever you think might be getting to or might already have archeology tech finished. You can use any available military unit to "occupy" an archeological site if you want.
About digging within another civs borders: You need to decide what your diplomatic strategy is. When you dig within another civs borders, they will approach you through diplomacy and ask you to stop taking their cultural items. If you oblige to this, no diplomatic penalty incures (unless you break the promise, obviously). If you refuse to follow their request, you will get a diplomatic penalty. However, the penalty is not that big and will be constant no matter how many further sites you excavate, so depending on circumstances, it can be worthwhile to take the diplomatic hit and continue digging - especially if you have finished exploration and they have many hidden sites within their area. Notice that hidden sites will count exactly the same as normal sites, even if the AI can't see the hidden sites themselves.
If you dig an archeological site in another civs territory, you get the option to make an artifact (goes to you) or you can make a landmark (goes to them). Making a landmark within a foreign civ or city state territory will give you a diplomatic bonus / influence boost (+40 influence with CS I think). I think most players don't use this option too often, but a marginal strategy can be to finish exploration and then dig hidden sites, and then leave the landmark unless you find one of the great works of writing, which you will obviously convert to cultural renaissance for massive culture boost. This way you can scavenge all hidden sites without getting diplomatic penalties, although of course you will give the AI the benefit of the landmark (unless it's outside of their workable borders, which will still give you the diplomatic boost).