While mechanically moving from builders to "point and click improvements" is almost negligible difference, there's a lot that having builders on the map does for gameplay.
It's similar to having religious units vs "spend faith to pressure another city."
Getting rid of builders removes a certain gameplay loop that fits well with the things going on in the early game. Otherwise we would just have cities, and some military units sitting around during peacetime. It would be really boring. There's a fine line with making it tedious line traders, though, although I thinkj civ6 does it okay, assuming the game starts in the ancient. Starting late game and improving huge areas all at once is a bit annoying.