How do you mean, losing military building bonus?
The buildings apply a malus to Military Unit Production, yes (usually -15%), but they also apply an EQUAL bonus to ALL production. Meaning your military production stays the same, and building/non-military production is increased. No penalty at all.