I let some build a trade network. They will rush and build improvements should new resources pop up in your cultural borders.
I do not see how it is a waste. You could disband them to safe the upkeep for having them instead, but if there is nothing better to do then building forts and running around then by all means... You will need them soon enough again because the late game improvements that are very good like wind- and watermills are taking a long time to build. You better have those workers ready by then! If they have nothing else to do in the turns until then then that means you have enough workers. Good for you and a job well done