I'd agree with you completely *if* the original rebuttal to any complaint about lore, continuity, physics, biology actually was simply "meh, it's fiction & it doesn't bother me." In other words, a simple expression of perspective & how different people have different "breaking points" when it comes to suspension of disbelief.
Instead, it's much more often the example you gave: "it's got lightsabers/dragons/magic wands so your particular complaint is invalid & you should be fine with whatever." It's not even acknowledging or addressing the actual point being raised, or allowing that different people have different breaking points when it comes to fiction. Instead it's a blanket dismissal of the point being raised as well as telling the other person how they *ought* to feel. That's why it generally triggers an equally dismissive & hyperbolic retort. All IMO of course.
I guess my perspective on this is that it seems to usually, or more often be the other way around. The first complaint is sometimes
technically correct/sound, but is also trivial, or petty nitpicking, in the eyes of the second person, and more importantly,
seems at best, arbitrary, compared to other aspects of the story that are equally, or oftentimes, far more absurd, fanciful, incredible, etc., and the complaining person hasn't offered
a reason why this particular concern is somehow more important ie
why this particular issue is the breaking point. The most the complainer typically comes up with, is the bog-standard response of "You introduce/set the fantasy rules in advance/at the beginning of the story for the audience then everything else MUST adhere to real world parameters", which has always been unpersuasive to me, since that's not how it ever works at all, in most fantasy/sci-fi/superhero stories. They change the rules/powers all the time. For eg., they don't show anyone using the force to move objects with their mind until ESB (Obi Wan didn't use it to turn the levers to shut the Death Star Shield down for eg.), so that power wasn't introduced at the beginning... force powers were always a moving target. So the "introduce the rules at the beginning" argument is just a rhetorical convention/rule, that complainers made up and typically fall back on in these types of debates. It does not at all explain
why the race of a stormtrooper is more important than the physics of spaceflight to them.
So then, the person complaining who is across the breaking point while the other person is not,
feels like their concern is being flippantly dismissed, and reacts with hyperbole "Oh so ANYTHING goes?!" But the problem isn't necessarily just that the person dismissing their concern hasn't given a reason, its usually also the case that the complainer hasn't properly justified
their own reason for their gripe, vis-a-vis anything else "unrealistic" in the story, so the person rejecting their concern really does not have much choice but to sound dismissive. A complaint I heard often about the the Star Wars ST from white guys was that the main white guy characters were all bad guys, while the main non-white-guy characters were the heroes, and this bothered them, because they 1)felt vilified as white guys, and 2) felt like they had no hero to identify with. So they dislike the whole ST for that reason. Now THAT is
a reason to dislike the stormtrooper being black. Obviously I would debate against that view, but it is an actual reason. Hobbits shouldn't be black because "genetics" is getting a hand waive from me, because "real world" genetics are already being wildly violated by the existence of hobbits, elves and orcs in the first place. So I need a better explanation why you (the royal you) don't like black hobbits. The dragons in GoT/HotD don't look anything like each other despite being relatives and nobody raises a peep about that violating any "genetics" rules.