I agree with the AI being crippled in their techs, and it might have something to do with the 25% build rate to improvements that has been added in FF.
I can see why this would be an issue.
Personally, I'd advocate the simplest possible solution. All workers should follow these rules
1. Workers should only build things they can do at 100% rate, unless there are NONE of those to do, anywhere in the empire. ie, every tile they have the tech for is already improved, or they have no improvement techs.
2. Whenever a civilisation discovers a new tech, all AI workers, (and automated human workers) belonging to that civ should immediately stop and re-evaluate their current task. If the new tech has allowed them to create an improvement more efficiently, and they're currently doiung a 25% speed build, they should stop their current task, and search for more efficient improvements.
This might put the AI at a slight disadvantage, as it would be unlikely to ever get lumbermills before Ironworking, and such things. But it would stop them from getting caught up doing things that take forever, when there are easier tasks to do. And would likely result in a better AI overall.
As an optimisation to point 2, it might be better to have a list of techs which allow faster improvements, and only make the workers re-evaluate if their civ discovers a tech which is on that list.
I think these techs are
Agriculture (farm)
Crafting (winery)
Calendar(Plantation)
Education(Cottage)
Mining (mine)
Exploration (road)
Masonry (quarry)
Construction (windmill, watermill)
Iron Working (lumbermill. I think this is too late, personally, but that's an issue for another thread)
This list only covers those which have a 25% build rate mechanic, not things like fishing boats, etc.
Did I miss any?