There are 2 types of early conquest:
- Conquering for land you actually want, where it gets you to an advantageous position for the rest of the game. (This is mostly what I do).
- Conquering for an early victory, usually with an early UU.
In the first case, plan out where you want cities. Their outer cities, without much infrastructure or population, raze and settle a new city in the perfect spot, often razing 2 cities to found 1. Their ancient cities, take, and raze anything you don't want that will have its culture overlapping your city on the 2nd culture pop, sometimes 3rd if the cities you don't want are already mature. (This is how you prevent cities flipping). The leader will probably hate you, so he's probably a target for elimination, not vassaling, unless you want his spies harassing your conquered cities. (This also solves the "We miss our old nation" unhappiness).
But it sounds like you're in the second case: Conquering for early victory. In that case, give everything back unless the city pays for itself. Gold or mature cottages? Sure, keep it. Killer production, already up to size? Sure, keep it. Anything else, give it back or raze it.
If you're giving it back, whip it into the ground for more cavalry first.
Also, if you go into anarchy, you stop losing money. One early Deity victory writeup was "Marathon, Pangaea, Julius Ceasar, whip Praets, whip every conquered city into the ground, civic swap for 3-4 anarchy every 5 turns."