I proposed something similar on the Apolyton forum.
http://apolyton.net/forums/showthread.php?s=&threadid=53003
Emigration, not culture flipping
My idea was similar to yours. There was little response, some agreed, others didn't.
My culture flipping proposal is:
1) Increased unhappiness in the affected city including a pop up window saying that those citizens admire another culture a lot.
2) After N turns, more unhappiness with another pop up window. This will usually result in population loss when entertainers are created to keep the population happy.
3) After N turns, one citizen leaves the city and becomes a new unit - a colonist - and appears next to the closest culturally better city of the different civ. That colonist can either join that or another city or start a colony, BUT it can not be a slave worker. This would be something along the lines of immigrants leaving to go to the 'new world' and create a better life for themselves in a different country.
4) Continued emigration until the last citizen leaves and turns off the lights, abandoning the city.