There are two defects to only improving enough tiles for your citizens to currently work:
1. When your city's population grows, the new citizen will have to work an unimproved tile, which will always have a lower yield that its improved yield. Even if you have an idle work sitting on that tile, ready to begin improving the tile, it will take many turns for the improvement to be completed. More realistically, your workers are engaged elsewhere (maybe in another city), so improving that tile will take many more turns.
2. You may want to change the focus of your citizens in a city (say, from food focus to production focus, or whatever). When you change focus, some of your citizens will be reassigned to work new tiles -- if those new tiles are unimproved, see 1. above.
An important variable here is worker travel speed -- it is highly inefficient to shuttle your workers between cities to improve one tile here and one tile there -- you burn as many turns with travel time as you do on improvements. Far more efficient (subject to the needs of your other cities) is to improve enough tiles in a given city to accommodate some degree of population growth and focus-shifting before moving your worker(s) to a new city.