Score is only determined by population and territory (and a bonus if you win before 2050 A.D.)
So you could have a large territory and high population, but you don't have the infrastructure to do anything with that. It sounds like you are doing a good job at expanding (grabbing territory early in the game), but after you expand to a certain point you need to switch to more infrastructure to make your cities more powerful. Decide early on what victory you want to go for, and start building/doing things to steer you towards that victory. Without a save game to look at or more details, I can't really say anything more.