This was a design decision by Hydro. I did disagree with it at the time but most building lines were his part of C2C. It is part of the idea that you can build stuff due to the technology you have but it has uses that only become available later. In this case you can build huts way back in the stone age but you can only have a bicycle shed when you get bikes.
There is no ability in Civ IV to allow a resource to be available only in the city it is in. In C2C vicinity tags and the like make it possible but those did not come until long after. In fact the ones which could be used for this did not become available until this year.
Having a building that gives the yield return to the city and then a replacement building that makes it possible to trade the good might work but then you would have the same confusion.
Well Cloth issue is by far worst one - Weaver Hut/Cloth unlock at Weaving - it can be your third or fourth tech you can research and Wool - first material you can make cloth from is accessible at Animal Husbandry, this is where you can actually build Weaver's Hut.
This delays Rope by 3 columns.
Essentially you have use for wool and fiber from silk/hemp/cotton/flax unlocked much earlier than those resources themselves.
What else is suitable material for cloth?
Milk exactly mirrors situation:
Theoretically you can have it at Animal Husbandry, but you can't make it until Glassware (X31), where you can finally actually build Dairy Barn after you build Glass maker or something.
It delays Cheese by 2 columns and Ghee Oil by 6 columns.
Just like can't carry milk in barrels/pottery and then use it immediately as food or make cheese/ghee oil from it.
That is you have uses for milk before you can make milk.
For me unrealistic resource trading is least concern.
By the way I don't think situation with some resources was that bad, when Hydro developed building chains.
That is max 5 columns between resource being unlocked and being able to build hut.workshop/factory producing that resource.
That is if resource was unlocked by tech, then you had to research less than 5 or so techs after discovering resource to build factory/workshop/hut producing resource.