SirPleb -
The way I tested seemed to show that claiming coastal and sea tiles didn't count towards domination. By not building a couple of 3x3 cities, i could build 3 times the number of 1x3 cities on the coast that also claimed 3 coastal tiles each. Also, having cultural boarders expand over the coast while at the domination limit, didn't trigger domination. I can't see how coastal tiles could count towards the % needed.
It could very well be that coastal tiles count towards the total number of land tiles though. For the most part, the larger the landmass, the smaller the ratio of coastal tiles to landmass tiles. That would mean that the larger the landmass, the less % of land tiles that would have to be claimed for domination. That fits with both our testing results.
Another thing that I didn't test, that very well could have made the difference in your testing, is that the AI was given another city, allowing more land tiles to be claimed by you. In my tests, the AI always had a set number of land tiles claimed, and a set number of cities. Maybe there is a modifier to the % needed based on the number of tiles (or cities) that the AI currently has. That could also be the reason that I was seeing the sliding percentage, as the ratio between my landmass and the AI's varied according to the total landmass. The larger maps (about 5%), the AI had a significantly smaller % of the total landmass than on the smaller map (about 10%).
Whatever the factors involved, Domination certainly seems to be the most complex of the victory conditions to figure out. I think it is a good thing to leave in the GOTM's. On "builder" type games, it adds a risk/reward factor in the score (be safe and only claim as what I have now? Or risk domination and claim a few more spaces?), as well as rewarding those with a better grasp of how domination works. Without domination all the games end up being basically the same.