Depends on when you're talking about; laws and customs always changed over time (there were especially a lot of changes in the early years of the Empire).
Strictly speaking an act would not be "adultery" unless it involved a married woman; if she were unmarried or a widow it would be a slightly different offense, but nearly as serious and with similar penalties. Reacting against it was generally the responsibility of the paterfamilias; i.e. the husband of a married woman, or the father, brother or uncle (etc.) otherwise. Under ancient customary law it seems likely that the paterfamilias could kill either or both offending parties; later laws prescribed more specific but limited penalties such as heavy fines, banishment, optional divorce, etc. (There were also specific penalties for a ****olded husband who neglected to punish his wife properly.) After the Empire adopted Christianity, an adulteress might be forced to enter a convent either for a period or permanently.
Laws applied differently to different classes. For instance, slaves could not be legally married in the first place, and it would not be considered an offense for a male citizen to have intercourse with a female slave (while a male slave caught in the act with the lady of the household would probably just be killed). And as ever, laws were often ignored or circumvented by those with the power and influence to get away with it.