Yes, you can lose your pantheon, if it is replaced by another religion and, frankly, even if it is not replaced by another religion.
Let's say, for example, you have a 6 pop city with 4 pantheon followers. Some neighboring cities with a different religion start pressuring that city. Over time, they get 2 of your citizens to flip to their religion -- 1 pantheon follower and one atheist -- and then your city rises to 7 pop, with the additional pop being a religion follower. At that point, you have 3 pantheon followers, 3 religion followers and 1 atheist -- neither your pantheon nor the new religion are majority, so the city gets no religion bonuses. Over time the religion will continue to attract conversions, while your pantheon (which exerts no pressure and cannot convert religious followers) continues to dwindle.