The thing that some people haven't mentioned is that you actually have to compare TWO towns w/all the bonuses against ONE specialist + one farm to feed that specialist.
Not true, unless all your cities are happiness limited. Many, even most, of my cities will be tile limited in the late game. A few of my biggest cities will have happiness problems in a war (before Police State) but they're easy to solve without wasting a lot of commerce through the culture slider. In Slavery whip to a smaller size, with Caste System starve to a smaller size. I aim to work every good tile under my control. So in my games 90% of my farms will support a specialist and if I had a town there instead of the farm the city would just be one pop smaller (and give me a lower score). So I contend ONE farm and its specialist is set against ONE town, if you design your SE properly.
That imo is why CE > SE later in the game. Once PP and all the CE civics come online 1 specialist just cannot compete with two mature cottages.
That doesn't mean of course you can't win with a SE--you can. But I am convinced that SE > CE early game and CE > SE late game.
That's because you're making a false comparison and possibly don't run your SE properly or it could be you don't build your cities in the right places to take account of what will happen in the late game wrt happiness. Having more but smaller cities is the way to go with an SE in BtS rather than fewer larger cities. That lets you whip or draft more efficiently and lets you build more EP buildings which means more EPs for better spying.
Cities in a SE fall into two categories in the late game. Either they can produce another GP before the game ends, in which case take care of their GPP pool and they need special consideration; or as is the case with most cities they won't be able to make a GP so you can ignore that aspect and whip and/or draft and convert their farms to workshops or watermills.
From that perspective I believe a transition economy to be the best, although it can be very difficult to implement. Basically, for a space race you want to be expanding your empire early, which means a lot of production (whipped or hard) which cottages do not give you. You also want to be regrowing after whipping faster and once you settle down into a transition you want to grow to max size quickly. Later in the game you want to be at peace because late game wars can hurt your economy through WW, pillaging, loss of trade routes, etc. etc. etc. not to mention that supporting a huge late game army is costly as well (better if you can get by with good diplo). Once you are at peace you are better off working max commerce because as most people will tell you the space race is about tech and less about production.
Transition economies can work if you are going for a Space Race otherwise they just lower your score and slow you down. Sticking with SE and running it properly is often just as powerful as a CE in the late game although it depends on the map and opponents as well. WW needs to be managed by keeping wars short, keeping cities small and as a last resort using the cultural slider. That is until you get to research Fascism and then Police State lets you war indefinitely and a SE should do that against any of its CE rivals as they hate offensive wars (which increase their WW and they need US instead of Police State to keep their production up). An SE gets extra production by converting its farms to watermills and workshops and by drafting.
So Late Game (post Biology, Communism and Fascism) a SE is the ideal war machine. In peace you aim to use Representation, Nationhood, Slavery (or Caste System), State Property, (any religious civic or FR). To go to war just change to Police State as soon as the WW gets bad (or if you’ve already fought this opponent change before declaring ) and change the religious civic. If you want to use Corporations then use FM instead of State Property. The UN resolutions can be a problem for your preferred civics but you and your vassals and colonies should have a lot of votes and you can always defy the UN resolutions (at considerable unhappiness).
Early game you also have the SE civics available (except representation in most cases and if you do get pyramids that is a bonus) whereas late game CE civics come online.
If you are going to run SE late you need to prioritize biology, constitution, and huge cities to run lots of specialists. Getting the food-based corps, esp sushi, is also probably a good idea.
Fascism is the most important late game tech if you want to go to war.
Another thing about SE late is that in caste system that means you are going to be dealing with emancipation
which of course is possible, but can be annoying and can at times require high investment in the culture slider (which can be ok, esp if going domination).
Emancipation is a pain but can be dealt with by whipping or starving and the small city strategy. Your few big cities, which are producing GPPs, need to invest in happiness infrastructure including multiple religions and temples and cathedrals or maybe Free Religion depending on what will give the best happiness / output. As a last resort you can always adopt Emacipation yourself until the happiness problems are solved but I hate doing that and considered I’ve failed in running my SE properly
On prince level and below you should definitely be looking at settling instead of lightbulbing with your great people (some would argue settling is better in higher levels as well...i would say it is situational at higher levels). This is because you will be self-researching most of your tech because the ai techs slowly on lower levels (if the AI is outresearching you, you need to revisit your economic management).
In BtS you have Espionage to level the playing field in the late game, even if the AIs use a CE to research fast and trade among themselves. A SE can use the Espionage system much easier and more efficiently than a CE can. If you have more (but smaller) cities as I suggest and each of those has a courthouse, jail, intelligence agency and maybe a security bureau, you can run up to 7 spies per city. With Nationhood that’s 49.5 passive EPs from the buildings and each spy will give 9 EPs for 2 food. Most of your cities will be giving between 50 and 100 EPs per turn and that makes it easy to see what all your rivals are doing and what they’re researching and you can research other techs and steal the ones they’ve researched. You should never be behind in techs for more than a turn or two as long as you’ve got spies in your rival’s cities waiting for them to finish research.
To the OP: a SE shouldn't just be
surviving in the late game it should be
thriving 
