No, I mean the farm itself (which you used precious charges to get) is not a good tile to work. In other words, with your first 3 charges you want to improve the first 3 tiles the city is immediately going to work as it grows (i.e. I would rather spend my charges on horses or sheep rather than an empty flatland farm for 3f1g or 2f1p1g). If you work the unimproved hill banana for example, then why waste a charge on farming a blank grassland tile when it's not a tile you're not going to work until your city grows to size 6 or so? (i.e. there are 5 other unimproved tiles around that are better than that farm, but thanks to housing issues, you kinda have to).
With one builder, assuming you make 3 flatland farms with your precious first builder (which is a disaster for production...) you get +4.5 housing, but I'm working other tiles which are worth working more... Let's say I have a generic vanilla civ with the exact same starting location, I get 3 from freshwater by default, then I improve one each of horse, sheep,and fur tiles. I'm still at 3+1.5 = 4.5 housing but my 3 pop city is outproducing its Mayan counterpart by 2 production and 2 gold. And on top of that, I got horses and furs to sell or gift to befriend AI early...
Also consider the early game tech path.... archery lies beyond animal husbandry... whereas irrigation is on a completely different path. The civ ability encourages you to go the pottery route instead, but If you don't beeline animal husbandry, you risk getting wiped... If you want to abuse their early science then you need to research writing ASAP as soon as irrigation is done (and on deity, I would think you are pretty much dead with just slingers at this point). Also, if you plan to do early war, being pretty much forced to start with a builder or face housing issues is terrible when you want to get that first slinger out and go hunt for that archery eureka ASAP! (which is why I compare them to Mali... but Mali at least gets that extra food and faith from the start whereas she has NEGATIVES coming straight out of the gate which do not get compensated for until ~20-30 turns later)