In my current game I have stayed with my first city and am trying to build buildings and wonders, and yet it seems that everyone else is shooting up the techtree and building higher wonders than I am, even though I've not wasted time building an army or anything.
I only have two workers in my city, should I have built a lot more before building buildings and wonders?
What am I doing wrong?
Emphasis mine.
This is exactly the problem. I have finished a Rome game on Emperor, and the thing about Rome is that early on, they're a war machine. The Legion and Ballista are your bread and butter: they're much stronger than the swordsman and catapult and only slightly weaker than the longswordsman and trebuchet respectively. If you're not going to use those, you're wasting the biggest advantage Rome has over other civs in early play.
No matter what victory type you're going for, start by building an army. Of course you can try to pump out an early Great Library and/or Oracle for their tech and culture benefits, but for the rest you should mostly try to conquer other civs. If you get a good iron source, you can conquer two other civs with a decent group of legions and some ballistae.
After conquering another civ or two, you should puppet their cities and you'll notice that your culture and science output have significantly increased. By the time you enter early renaissance, which means your UU's are obsolete, you can settle down and truly focus on science.
So in short, I'd advise you to do the following from the start of the game:
1. You won't make it with Rome alone, their UA depends on you having other cities. Try having 2 - 4 cities
2. Beeline ironworking for your legion, and then research mathemetics for your ballista
3. Start looking for an iron source. If you don't have any, found a city near a good source
4. Start pumping out legions and ballistae. You can use the legions to connect your cities until your army is built up to a decent size
5. Start conquering a civ or two, puppet their cities
6. Make sure your happiness stays at a good level through luxury resources and buildings
7. Meanwhile, make sure you have all important buildings built in Rome so other cities (including puppets) will have +25% construction speed
8. Rinse and repeat until the renaissance, after that start pursuing a science victory with a much larger science output
Also, make sure you construct the National College ASAP in Rome.