Why is it still a problem now?
I'm curious as to how a problem in Civ 5 would have been fixed 8 years ago, seeing as Civ 4 wasn't even released back then. You also seem to be making the assumption that the end problem now is caused by the same issue as it was in previous Civ incarnations. There is no evidence to indicate that it is, and the fact that there were far more workers on equally sized maps in Civ 4 without a matching increase in turn times indicates that it wasn't, and as such your impotent nerd rage is pointless (thus the aforementioned impotency of it).
The problem stems from workers evaluating what to build. When a worker is selected, either by the player or by the AI, whether you have improvement recommendations enabled or not, it will evaluate possible improvements to be made to all tiles within the cultural borders. This causes two problems: first that selecting a worker on a large map with a large empire can lead to significant delays (upwards of 30 s), and second that processing the AI turns can take an inordinate amount of time as each worker individually processes the potential improvements to be made to the entire empire (and unfortunately the AI likes to build lots of workers).
The fix is almost certainly going to be just caching the information (warning: amateur armchair game design theorycrafting ahead): calculate it once at the beginning of the civilization's turn (AI or player turn), update tiles when they are obstructed by a moving unit or another worker calls dibs on building an improvement there, and update if new tiles are added by founding a city or culture bombing. For the player side, this will move the calculation from selecting a worker to the time it takes to start a new turn, but it'll only be done once instead of each time a worker is selected and the increased workload from the player's calculations should be easily offset by the reduced workload for the AI's calculations.
Of course, getting multithreaded AI to calculate this exact sort of information in the background while the player farts around deciding things during their turn would be the best option.