It's annoying when what your city needs is housing or a district, but there's nowhere to place any neighborhoods or farms or anything because there's a whole bunch of Luxuries (dye, silk, etc.) scattered next to your city. The tile yield is not very good on many of these tiles. After the first copy, they don't provide any benefit unless you can trade them. The player should have the option to build a neighborhood or farm on top of the resource and remove the luxury resource. In the late game it would be especially useful. Also, in the early game, sometimes the luxuries completely mess up the farm adjacency bonus when farms would be much more useful than a measly plantation.