This is mostly decided in the CvCityAI::AI_buildingValueThreshold SDK method which is over 1200 lines of code long, so it's not something that can be easily summed up in a few sentences.
It depends on many factors such as the happiness / health of the city, what type of victory the AI is aiming towards, what the city itself is focussing on, whether you are at war, whether the building will improve your commerce or not, what religion you are running, whether you have the holy city for a religion, what your civics are, the list goes on...