Keep in mind that the gold or science that you build does not go through the modifiers. For example, you are building science in a city with a library but the beakers converted from hammers do not get the +25% library science bonus. Building science or gold is generally inefficient I rarely do it except when a) I am desperately broke, or b) there is really NOTHING else to build.
City specialization, while crucial to success at higher levels, will never be perfect. For example, I often find myself building a barracks early in a city that I know will be a specialized for science because a) I am fighting an early war and simply need the units, b) it is in a threatened position and may have to whip units later, or c) I am aggressive and the bonus makes it real cheap. Many science cities, if they have reasonable production, can be unit builders in the downtime between new science buildings. I don't think there is anything wrong with a little hybridization along these lines so long as it is not hindering their main purpose which is to produce science.
The real key is to distinguish science from gold cities. I go for roughly a 2:1 ratio in favor of science cities. The gold cities will focus on building markets, grocers, and banks and the science cities will focus on libraries, universities, and observatories.
Assuming some use of the slider, though, there will always be some crossover. For example, any city that generates commerce will produce some science and some gold and thus, can benefit from both types of buildings in the long run. The main reason for early specialization is because of the amount of hammers it takes to build those buildings. You cannot have them all in every city or you will be wiped out by someone who specialized, built only a few key improvements in each city, and spent those hammers on military.
Like DarkFyre I consider courthouse and granary as key infrastructure in every city but I usually hold off on forge except in production cities. Anything important can usually be whipped.
Sorry for the long response, hope it helped.