There is a limit to the effectiveness of luxuries, depending on city size. Luxuries only work up to twice the city size. Your size 4 city only gets the use of 8 of the 13 luxuries it produces. A size 5 city would get the use of 10 luxuries, etc. To eliminate the unhappy hat, you will have to build a temple or other contentment building.