I view video game design to be about as creative an endeavor as producing a film. And while there are arguments to be made about how film schools and Hollywood make a definite effort to weed out creativity and imagination in filmmakers, there are still plenty of directors, producers, and other individuals that work in the film industry that get to express their creative intent through the medium. Arguably, many of the best movies are the ones where that creativity and imagination get to shine through. Similarly, I'd say video games are best when their designers get to express their own creative vision and intent, rather than being forced into a mold of making a product aimed at catering to the market.
I'll agree that game designers are caught between multiple competing demands. I do still feel that most people that go into creative industries do so because they want to bring their own ideas to life -- I've known plenty of artists that got frustrated at the constant parade of individuals who come by asking for someone to make
their ideas a reality, rather than those the artist wants to do. At the same time, the people that run the companies, whether they be film studios, publishing houses, or video game companies, want to make money, and the most effective way to do so is to cater to what the market demands. And so those designers, writers, artists, film directors, and so on, are typically caught between their own creative desires, and the demands of those who handle their budgets.
So I'll agree that people that produce creative content (whether books, movies, video games, or whatever) have to pay attention to what the public wants in order to maximize profitability. Exactly how much any particular producer wants to emphasize profitability vs. other priorities is up to them, however. I know of at least one privately owned gaming company (a tabletop gaming company) that, due to being privately owned, can choose exactly how much they wish to cater to the public to maximize profits, and how much they can follow their own creative interests. They've used the profits from their main game to produce several side games (of varying quality) that they wanted to make, even if their primary fanbase didn't really ask for them (or even want, in some cases). But since it's their money, they can decide what to do with it.
In the case of a larger corporation like Firaxis, they likely have shareholders to answer to, and thus have less freedom in deciding to eschew profits in favor of vanity projects or other, more creative endeavors. But at the same time, that still doesn't mean they'll be committed to providing what certain vocal parts of the market demand. There are plenty of productions that ultimately cater to the lowest common denominator and the largest markets -- look at things like the Twilight films, or Konami's move away from more traditional video games towards mobile games. There are enormous markets out there that can apply much more market pressure than a few hardcore, traditional gamers can, and thus having a company focused on profitability over other priorities (such as making a high-quality game, or one that has a steep learning curve as some games had in the past) can ultimately work against what some in the gaming community would desire. Look at all the complaints about how companies are "dumbing down" games in order to appeal to the more casual gaming crowd with games that have less barrier to entry.
So you have people that end up arguing that companies should prioritize "making a quality game" over profitability, but then those some companies can choose to put their own "creative vision" over creating a game those particular people desire.
In the end, probably the main reason I dislike the whole "the gaming industry exists to serve the desires of the market" argument is because it is one that reduces agency and free will -- it's based on a philosophy that other people exist not to fulfill their own goals, but to serve the will of the person speaking. It's not particularly egalitarian, and often comes across sounding like indentured servitude. "You're not allowed to make the game
you want... Your entire existence is to serve me and make what
I want." In the end, of course, reality doesn't work that way. The people in these companies are all out there seeking to fulfill their own personal goals, whether that's to fulfill a particular creative vision, or to line their own bank accounts with more profits. "Serving the public" as an altruistic motive is probably pretty low on their list of priorities. Which is why I suspect those that feel those people who make entertainment products exist to satisfy their personal desires are ultimately just going to end up feeling frustrated that no one is particularly listening to them... and hence, why it's a rather foolish philosophy to follow.
In the end, I would argue against the idea that "companies should listen more to the market", and more that "people should stop expecting others to provide them with what they want, and instead work towards it themselves" -- a philosophy that is more self-reliant and less dependent on others.
And as for whether or not BE was a success... I would determine that by how many of the goals that the people involved in making it set for themselves were achieved.
It's understandable of course, but it leads to mistaken assumptions which pervade these kinds of topics.
