I think I would feel better about it all if the Infernals were given a Ritual they could perform, per city, to convert the citizens into Demons (thus building the Infernal-only structures). It should convert some proportion of the citizenry to infernal life (i.e. you should knock some population points off to represent the people that could not be converted/whipped/encouraged to become demonic), rather than a full 100% swap.