I think it's caused by getting heroic relics for heroes that you didn't recruit. It started happening to me when that game mode was added, anyway, and now I avoid buying heroic relics from the AI and I don't seem to have this problem anymore. It's possible that something else has also changed, though. Maybe a mod or something...It’s one of the strangest bugs in the game. I have absolutely no idea what causes it but it frequently affects me too, mods or no mods. In fact, I have to use a mod to fix it (Better Great Works Viewer or something like that)
Anyway, there's also a bug with using Cultists to generate artifacts. The game puts them in places that don't have room instead of putting them into empty slots. It just eats your artifacts.