It didn't have that many viewers because it was in a time slot right after a sports game, also the fact that the pilot and the first episode were shown in reverse order confused some people. Since it was in that time slot sometimes you'd miss 15 minutes of the shown due to overtime...
This is basically true. But there's more to it than that.
Fox screwed up. They took a show that would most likely have had a solid, but smallish, loyal audience and they did not support it to the needs of that audience. The audience was probably never going to be large. Whedon shows just don't have that appeal to large audiences. Then Fox interfered with the creative process too much. Which further narrowed the show's appeal. Beyond that, they supported the show extremely badly, showing it out of order, putting it in a time slot where many of the likely viewers would not be at home, and beginning the show when it would be interrupted by baseball.
The mistake Whedon made was to have his show on Fox at all. In that time period, Fox simply had no patience for scifi/fantasy/quirky shows. They were very unlikely to give the show time to find its nitch.
I don't have the sources to prove it, but there was a lot of discussion a few years ago that the business model Fox, and much of Hollywood, uses for which shows to create, continue, and cancel (movies as well) is deeply flawed. And while I've since lost the sources that people were using to make those arguments, it seems to make a lot of sense. For one thing, as the market was becoming more globalized and fractured, the analysis of movies and shows was based only on US income. So much of the potential income stream didn't go into the calculations.