In my experience alienation of casuals and silents comes from constantly seeking to perfect balance.
Much of the community push for change comes from a small group of players (those active on the forums) who seem to prioritise balance over anything that might be OP. IMO these players would prefer a game where every start is equally good, every policy is equally good, every belief is equally good etc etc etc. (Obvs with the caveat that all decisons is situational... terrain, neighbours, etc). One of the goals of these players is to eliminate anything that might be OP.
Over time that leads to eliminating fun. Especially outrageous OP combos that were super fun and a touch broken. (This is the part where I wish I had a better memory cuz I can hardly remember any of the specific changes that felt that losing fun... maybe nerfing Oracle, various founder beliefs, Poland, etc etc). I've always felt like if you want to play a game without taking advantage of the AI by picking OP things... just don't pick those things! Don't eliminate them for the rest of us who enjoy them! ( and if you dont want the AI taking advantage, go to a lower difficulty, or use it as a handicap).
So I'd say that those who care most about VP ( as expressed by their forum participation) have shaped the mod to be what they want: a deep, involved strategy game where, in a vacuum, all choices are equally advantageous. That is their right, they obviously care the most. But for me and some other players, the mod has lost some of its fun. It's lost those golden moments when BAM! some combo of 3 or 4 things slingshots you free of the grind.
I still play the game* but it's become less fun for me. I appreciate the constant churn of versions and hunt for a better mod, I just think the game experience is a certain kind of less fun. And I suspect a lot of silent players would prefer a mod where there is a little OP here and there. To me thats where the alienation, such as it may be, exists. Not everyone wants a perfectly balanced mod where hundreds of games can be played without having to worry that you or the AI could win b/c of unbalanced design. Some folks just want to have fun, play a couple times a month and smash a glorious game-breaking combo or two (while still enjoying elite tactical AI, proper diolomacy, etc etc etc, all in one mod!)
Edit: another way of putting it is that (winning/succeeding at) the game is now almost exclusively about attrition and accumulation to the exclusion of the kind of 'fun' I tried to describe. When all points are equal you have to just try and get as many points as possible as fast as possible. Still fun but missing those pops.