Keeping your two closest neighbors as your friends is the key here. (Quit right away if it's Ragnar for one of them). Ideally you get a religious AI on one side and a reliable, don't-declare-at-pleased, non-religious type on the other and the three of you form a religious bloc. If that happens, you don't need much more than archers for almost the whole game, as your two friends will be the ones warring. And by the time your enemies can get a stack close to you, you will probably have machine guns, artillery, and all that good stuff.
Good advice. I recently went back to a game that was going well before I was dogpiled and did some things different to prevent conquest defeat, and came up short on the space race (actually abandoned when I lost the Internet race by 1 turn). My best game was with the religion block, before Mansa rolled all his neighbors taking half the world and started teching things like Fusion in 2 turns.

I was tooo embarassed to mention I added Mansa because if he gets big he is a serious competitor for space, and I thought maybe this was my mistake.
As for religions... as result of CS-sling I tend to have Confu HQ, which deters AI from sending missionaries (even if I don't adopt it) at least untilö AP is built. I can sometimes get one neighbor to go confu, but more often than not both neighbors already converted to something else. Catching a religion is made difficult by the Confu HQ... maybe I should just skip the CS-sling?
I'm playing on high seas to limit AI size. I've tried every climate variant (Cold/Arid/temperate/tropical/etc) to see if bad terrain can slow the AI, but it doesn't seem to matter. Max or min opps? Around the Industrial age one of them ends up winning a war and gets big no matter what, at least with vassals turned off. With vassals on, I get very close to being Diplo-loss (but I haven't actually lost one this way yet, so I think I leave vassals on for the next try).
By the way, you say you either A. get dogpiled, or B. lose the race.
My reaction to A is to discard games with borders to a warmonger, and to add Mansa and Sitting Bull always, as they will draw a lot of fire, while being easy to deal with (and why wouldnt you have Mansa to help you tech faster anyway?).
As for B, If you can get an early tech lead and not keep it, then you are either not focusing enough on lategame beaker output, or your starting location is too poor.
Did you try saving forests (10+) and beelining biology? Did you get that NEpic+GLib early enough to have a LOT of settled GS?
Also, one point I learned the hard way, not starting on a river is a pretty big handicap.
Yes, after a number of attempts trying to replay a game that gave me excellent land/resources but had Toku as a neighbor, I have come to same conclusion on abandoning games with bad neighbors. Mansa is good for making faster finish, but danger for having him win. But I suppose there might be some tricks to prevent him warring his way to an enormous empire?
I suspect my starting location choices are not optimal. I always take river (levee, early health/commerce bonus), and have been favoring 4-5 flood plains that have forest to balance, and usually a gold and food resource or two.
I have not tried pushing to biology asap. I'll give that a go next time and see how I do.
It is very hard for me to tell what is simply bad luck and what is poor decision-making, esp when things seem to be going well before they all fall apart. But I'm learning. Thanks for the advices!