Scaling up individual interactions to understand larger systems is always a dodgy proposition. Sometimes it works, sometimes not. It fails miserably in economics which is why the field of macro was invented. It fails in biology too--imagine if we took an understanding of single cells and said larger forms are single cells interacting in aggregate and then pictured their holistic being--we'd have quite a meat blob and not the superstructure that informs the meat blob's real shape and functionality (like humans). Sometimes we have to start at the top and fill in the pieces.
In your example, you are scaling up everything. There's now a giant family unit with lots of bad guys and a bunch of probably not bad guys, with a bunch of dead folks, but not lots of dead folks, and there's a whole lotta police serving an even larger police institution who show up after the fact, and then there's the observer who isn't any of the parties... it just doesn't connect. So if you make that your metaphor for nuclear retaliation, it seems crazy.
In the OP scenario, there are no police. There are no societal rules to attend to the attack. There's no time or mechanism for there to be a trial-no-trial dichotomy. There's no after-the-fact murder to attend to, there's group about to kill you and everyone in your group (as defined by them) and you can't stop it. But you can decide to take them and the people they are proximate to down with you.
How you decide, how you figure out what's moral, in the OP vs dealing with a murderer are not merely a matter of scale but of different categories.
For example:
In the police scenario, there's no family industrial/science/institutional base necessary for the murderer to murder again. There might be an issue of family culture, but not so much of requisite practicality.
There's no reason to think that now the murdered person is gone, the murderer will begin a campaign of world domination. But in the nuclear scenario, that's a real concern.
In the murder-scale world, the murder can act alone and with certainty one can deal with the murderer as an individual. In the nuclear scenario, the level of uncertainty is far greater in proportion of blamable-agents that scaling up the numbers.
In the nuclear scenario, and here's where Mouthwash and Akka weren't terribly far off in asserting game theory, there is a cost-benefit issue at stake with that uncertainty. How many more, after you're gone, will they kill or threaten to kill? How many of them need to be stopped to prevent that? How much comes from their roots and will re-grow if you merely take out the leadership or military installation? All the factories, the technology, the labor, the national psyche, the internal momentum, it's all there. If my country loses 100 million, and their country loses 200 million in my retaliation, and let's say 199.98 million of them are innocent, is it still worse than risking the remaining 6.7 billion people on earth to their reign? Or the reign of emboldened copycats?
Maybe you can play it safe--maybe you're the bad guy and after their strike, if you don't strike back, it's freedom and Utopia for all the billions of survivors. Han Solo shot first. But if they're the bad guys and you leave survivors, will they rise up and do more harm than if you killed every last one of them?