Well... I did no research on this, but simply played the game for some 17 or so years. I not sure of this, but I guess, there must be another factor that influence this aside of points only. Maybe minor, but anomalies happened. I had perception that sometimes I lose the castle parts progress on reload. For example I got castle part in, lets say 2400BC. Then I reload 2440BC, kill annoying barbarian thanks to reload extra turn, do all the same, maybe found a city one turn earlier, but not get castle part until 1900BC or so. Unhappy citizen? Maybe…
I've never really paid attention to castle construction, but things like above happened to much ever again to completely ignore and forgot them.