See attached graph of military power. I was white. I had one powerhouse neighbor (yellow) with another powerful civ on another continent (purple). Yes, purple won the game, but you can see that lowly white brought down mighty yellow before that happened and without having to build up a huge army. Yellow and Purple were not at war. Composition of the forces is, of course, very important. Yellow had no air force until I was in control. But my point is don't throw in the towel just because you have a heavily armed neighbor. Make your plans and be sure you are the one to start the fight.