Were the adulterers killed or imprisoned or was adultery a way of life and nobbody really cared about it? Or was it encouraged?

Depends on when you're talking about; laws and customs always changed over time (there were especially a lot of changes in the early years of the Empire).

Strictly speaking an act would not be "adultery" unless it involved a married woman; if she were unmarried or a widow it would be a slightly different offense, but nearly as serious and with similar penalties. Reacting against it was generally the responsibility of the paterfamilias; i.e. the husband of a married woman, or the father, brother or uncle (etc.) otherwise. Under ancient customary law it seems likely that the paterfamilias could kill either or both offending parties; later laws prescribed more specific but limited penalties such as heavy fines, banishment, optional divorce, etc. (There were also specific penalties for a cuckolded husband who neglected to punish his wife properly.) After the Empire adopted Christianity, an adulteress might be forced to enter a convent either for a period or permanently.

Laws applied differently to different classes. For instance, slaves could not be legally married in the first place, and it would not be considered an offense for a male citizen to have intercourse with a female slave (while a male slave caught in the act with the lady of the household would probably just be killed). And as ever, laws were often ignored or circumvented by those with the power and influence to get away with it.

So if it were a married man having a bit on the side with an unmarried mistress, that would not have been regarded as adultery?

So if it were a married man having a bit on the side with an unmarried mistress, that would not have been regarded as adultery?

Indeed not, technically -- adulterium applied solely to some guy (married or not) getting it on with another man's wife. It would still be an offense, the term used was stuprum which most readily available dictionaries list as "disgrace, defilement, ravishing, violation" etc. In most periods, similar penalties would apply, but depending on the situation the offense might not be prosecuted as diligently. In the case of a young unmarried woman, her father or brothers would probably be pretty ticked off; if it were a 40-year-old widow acting in practical matters as head of her own household the whole thing would be much more likely to be overlooked.

So if it were a married man having a bit on the side with an unmarried mistress, that would not have been regarded as adultery?

I thought Romans followed the old greek system.
Married men were allowed to have affairs with un wed women but not with married women. Such acts were dishonorable

Women on the other hand when married were not permitted any extra marital affairs of any kind. And must uphold the purity of the marraige

If I remember rightly, Caesar more or less publicly cuckolded his political opponents. On average it was only adultery if the offender was a woman.