Yeah especially in my current Indonesia game on large Earth TSL, my island is only 4 hexes. I can't afford to waste valuable production by settling on those diamonds. I had no choice but to move off into a regular rainforest hex and build my city the next turn. I needed every ounce of production I could get. At least until I built enough galleys to conquer Kyoto, sorry Japan, you were the only one I could reach since Australia was out of reach (until cartography).
Usually the initial square turns out to be a pretty good city for me, so I rarely move. Even when I have more room and production, I often still don't like to build on resources, if I can find a flat hex with water that I can move to and settle on the same turn, I often do.