I usually don't build it unless I have at least 2 or more quarries lying around.
That +2 gold per quarry can save your early game, tbh.
Two stone, one marble + 2 faith from quarries + mausoleum + Stone Works = well, extremely strong quarries.
And of course, the 100 gold for each GP is nice, but for me it doesn't make a huge difference.