It's very simple:
If you want Domination / Conquest: Stay in Slavery the whole game.
If you want maximum tech (i. e. Spacerace / Diplo / Sushi / etc.) : Switch to Caste + Pacifism
Apart from that, Caste + Pacifism are very nice in Golden Ages and the Border-pops from Artist-specialists are the easiest ever, so with being SPI, one might want to switch to Caste only to not build some Monuments ^^ .
Another good reason to run Caste btw. are that the Merchant-Slots are "cheaper" , because Markets and Grocers are very expensive. Therefor, one might want to switch to Caste System if needing a GM (i. e. for mass-upgrading Cuirrassiers) .