Well, a colony is just a resource extraction site, like a mining camp or lumber camp. While there is some limited economic activity at such sites, keep in mind that while large towns have grown up at some such sites, in others the city just dried up and blew away when the resource played out. If you want growing towns from these, then you should also have the city disband sometimes if the resource is used up. Do you really want to model things that way? Suddenly you get the dreaded message: "This source of iron has been used up!", and all of a sudden, where you had a nice little size 6 city with half a dozen improvements, there is only a settler, waiting to wander off to greener pastures.
Maybe when a colony falls under the domain of another Civ, you would have to pay gpt per turn, or part of the resource as tribute, in order for the colony to remain. Of course the other civ could have the option of ordering you to leave, or declare war. This could make for some interesting gameplay and negotiations, especially if a new resource system such as I and others have suggested is implemented, in which there is a set amount of the resource to be extracted per turn, available to store or make units/improvements, with the possibility of much more being available on the site discoverable through new techs. Here it would be useful to have the resource exploited through a special tile improvement that takes several turns to build; if you do have to leave, your people pack up all the tools and equipment and take them back home, leaving the other civ to take 10-15 turns building a new mine, oil derrick, what have you. This would make negotiations more attractive, as the out of luck civ would have time to consider military options for reclaiming that tile, and in the meantime, the civ taking it over would not be reaping any benefits from it at all.
Over time, it would be possible for the colony to gradually fall under the influence of the other civ, and you would lose it, gaining only a worker wandering back to your lands, representing your own loyal diehards; the rest of the colony workers having fallen under the other civ's spell through daily interaction with its citizens.