I think this was in another thread but someone suggested saving and quitting the game as soon as they start seeing the yellow text on the upper left hand corner (I think they say something about like memory is overloaded or something like that), then restarting the game. I also had major performance problems towards the late game routinely, to the point of freezing and crashing. But since using this trick, it's been working. However my hypothesis is that you have to quit right away when you see the yellow text - if you let it go on and on, your game will be screwed even if you restart.
However this is all just based on what i've done so far so i don't know if that's the cause. But maybe you should try it and see what you think.