It's a balance, but generally you always want to be working all 4food or better tiles. If your happy cap goes way up, you have to decide whether it is better to grow or keep working the specialists. Suppose you have a size 12 city, stagnating while running 5 specialists, and you have 5 irrigated grasslands you could work, and you have happy cap to grow to size 15. You will have to grow to size 15 before you can add another specialist without starvation. To find out what is best, figure out how long it will take to grow those 3 sizes and how many specialist turns you lose. That's how many turns it will be after you reassign the specialists until you get to the break even point and start getting more GPP than you would have by not growing. Then decide if it is worth it. On normal speed, the above example i think would take about 5 turns to grow to size 15 if you have a granary and send all the specialists into the fields. Thats 25 specialist turns ... so if you start growing, in 30 turns you will start seeing the benefit.
If you are going to grow a GP farm, it is best to do it immediately after getting a Great Person from that city. You don't want to be just a turn or 2 away and then fire your specialists to grow.