The big issue with a pop-size science model, is that farmers generate more science than science specialists over the long run if you use an N^2 pop model.
I think what you actually want is a floored linear model with a base penalty and a high constant. At a guess the first thing I'd try would be max(0, 2n - 5) where n is the city population. Size 1 cities make no science, size 2 cities make no science, size 3 cities make 1 science, etc..
This way every time you decide to expand early you do so at the expense of technological growth, and cities don't support gaining technology until they reach a sufficient size (in the example 3).
This enables you to nerf growth in a very real way, while still keeping value to science specialists. If growing from 10 to 11 results in going from 100 to 121 science per turn from population you just never make science specialists at all, but would instead go crazy with farms for science growth. Also the libraries and such giving linear bonuses would just never get built at all (your +50% bonus to base science instead becomes increasing science from 100 to 105 in a size 10 city, which is such a trickle its not worth the upkeep). I mean if you want to make a mod where the population based science amount is based on the square of the population I can pretty much guarantee the optimal strategy will be to expand your first city as large as possible and only expand outward when it doesn't stall your science growth (i.e. when your number of turns for growth is low enough and your luxury resources are high enough to settle more). Pretty much you'd want to always avoid hitting the growth penalties and grow horizontally as much as possible. OCC might even be easier in such a mod because it would prevent someone from expanding when they shouldn't, and India's civ bonus would be completely and totally unfair, they'd easily be the best civ in the game with even quadratic pop to science ratio.