I've been thinking about evacuation/abandoning cities ever since the topic came up in the Civil Wars thread. As Shyrramar pointed out to me over there, having to manually build workers to move a city's populance elsewhere is very tedious (in Civ 2 you couldn't just abandon a large city the way you can in Civ 3). But on the other hand, having all the people from a large city just "vanish" isn't very realistic.
The problem with refugee/immigrant units that are free to build, however (and especially if you can build more than one per turn) is that it become relatively easy to move a large city over to an adjacent tile in a short amount of time. Simply build a settler, then abandon the city or whatever you need to do to make all its people into refugees/immigrants, then settle them all into the new city that the setller build right next door. You wouldn't do it for one of your own cities, since you'd lose all the improvements, but it might be somewhat of an exploit in a newly conquered city, especially since you could send the refugees off to your core, where they'd be unlikely to cause a culture flip, and bring in immigrants of your own nationality to instantly have a large new city of your own. This sort of thing should of course be possible, since forced relocations have happened throughout history, but it shouldn't be too easy, and I fear that might be the result of population-units that could be instantly built/drafted.
Here's one possible solution: refugees/immigrants can only join cities that have a granary, and they deplete some of the food from the granary on the turn they join the city. That would limit you from building a city too quickly using these units: once the granary was depleted, you'd no longer be able to add them (you could still add regular Workers and Settlers) and instead you'd get a message like "This city does not have enough food for these refugees - they must look elsewhere" The amount that each refugee depleted from the granary would have to be playtested, but it should be high enough to keep from from instantly building large cities with these units, i.e., a couple units should probably deplete the entire granary. And of course, adding refugees/immigrants to a city would slow down how fast the city would grow by normal means, since the granary would have to get refilled first.
Hopefully, this would keep people from using these types of units exploitativelly to move people around at will and instantly make huge new cities, while at the same time allowing the concepts of immigration, citizens fleeing from war zones, and evacuation/abandonment of unwanted cities to be in the game. If Civ 4 allows food to be moved around from city-to-city, as many people hope, then these units (workers, too) should eat food each turn (as settlers/engineers did in Civ 2) to prevent the exploit of saving a city from starvation by simply making refugees and then having them rejoin the city once enough food was available.