I try not to build any settlers. Building early settlers stunts the growth of the capital and wastes a lot of turns which could be more productive building something else.
I use Collective Rule for the free settler and buy another one from selling early luxuries to the AI. I try to develop a 4 city core for my empire, and the 4th settler is sometimes purchased and sometimes built depending on game conditions. If I do build it, it is well after Collective Rule and after my capital has grown quite a bit, so the number of turns wasted with no growth and no infrastructure improvement is minimized.