I think there is more going on here than just losing the great library. If you want to kick this person's butt without going to war and literally kicking their butt, you need to build infrastructure quick so you can get your cities to a high population while they are messing around with a great library. The quickest path to victory includes the following in any order:
1. Build two scouts and get pottery as your first tech.
2. Build a granary and a worker, and a shrine if you need it.
3. Find a city state, get right next to their worker with your scout, declare war on it by moving on to the worker's tile, then click the city state and make peace. Do this only once as it will make all city states mad if you do it any more.
4. Build 2-3 settlers, and get them to build cities within 4-8 tiles of your capital. Preferably in a circle around it, but make it strategic.
5. Improve every resource you can, and wait until each city becomes population 5 before you build roads or they will waste money.
6. Build caravans after other cities build granaries, and use them to send food to the capital. Cargo ships are better, but you will need to keep pirates away or they will get them. Food via caravan is beyond important.
7. Keep an eye on happiness, and grow that capital as big as you possibly can.
8. Go tradition, only tradition, all the way until its complete.
9. Prioritize techs for resources, then construction, then education and finally acoustics. Open Rationalism and fill it until you get an ideology.
10. Get libraries in all cities and build national college, 4 cities is all you need so don't build any more unless you can't contain yourself.
11. Save up as much money as possible, and buy as many universities as possible on the same turn you research education. Do the same thing for Scientific Theory, and Industrialism.
12. If Gold is a problem, save it all for industrialism. You want to buy a factory in half your cities the same turn you get it so you are first to the ideology of your choice. Ideologies are amazing and help immensely.
13. Go down and grab Chemistry and Fertilizer, then head north again to Plastics as soon as possible and buy all the research labs.
14. Win peacefully, without a shot being fired, either through culture, diplo, science, or time. Or use the insane Technology advantage you now have to squish everyone flat. Either way, your cities should now all be close to 15 and capital at around 25, and you can pretty much do whatever you want, whenever you want.
I left a lot of things out, because this is just a bare skeleton skin and bones backbone to any victory condition, but it covers the basics of an easy Civ V victory. If you do these things, you will win. Its only a matter of how long it takes and how you want to do it.