We got 11 workers right now, assume we got 12 workers. With Serfdom, these workers gain 50 % efficiency. This means, only the upkeep for 6 more worker is 6 gold per turn more.
The Caste System is premature, as most cities have not grown enough to make specialists a feasible option. This is why serfdom comes earlier in the game, as we normally develop the land to the point cities get bigger, and caste system makes sense. Some citizens want to jump there right away, even if most of the land is underdeveloped. Daveshack and Methos did some analysis, and none of them supported Caste System.
Imagine if we had say 20 workers, then these would work like 30 workers.
Right now, the bonus means one to two cleared forest tiles per turn, with 20 workers, the bonus would mean 3-5 tiles of road per turn. This is volumes of economics. Also, Serfdom is a low cost civic, whereas Caste System is medium cost, with a difference of about 7-8 gold per turn. This means that the gold cost difference in upkeep alone is 15 or so gold, minimum.
Now add the advantage of adding hammers from chopped trees, cleared jungles and so on. Knowing Brazil, they would be in a very bad position for example, if they could not chop the jungle.