It's not a matter of fairness, rather a matter of difficulty. 1 v 1 is definitely way easier than 4 player map. It takes the AI most of it's advantages - rapid expansion does not work cause of little land, conquering weak neighbours that don't know how to defend and becoming a runaway does not work cause there is no one to conquer. Make alliances to take down human player from all directions don't work - these are all things that the AI does the best, staying small is only an option for the human player, the AI does not know how to win that way. In addition human can go for quick army and there is only one city to conquer to get a victory. Even in 4 player map, when you commit to an early army an conquer one of the opponents, the other ones might invest into wonder/tech and become runaways. On 1 v 1 you don't need to worry about that.
However when increasing the size to larger map a little further the corelation in difficulty increase isn't as simple, as some victory types and some strategies are more difficult on 4 player map than on 10-12 player maps. But 1 v 1 is definitely the easiest, which doesn't change the fact, that it is fair. As it is one of the options to choose from (just like policy saving or promotion saving, which make the game a lot easier).