I wanted a single list of everything that affects happiness. I couldn't find one, so I compiled it myself. If I have some details wrong, please post corrections and I will edit.
HAPPINESS
+X difficulty level
+1 each dye, fur, gems, gold, incense, ivory, silk, silver, spices, sugar, whale...