My sense is that that it has to be excess infrastructure, which I doubt any city has. To accommodate an Olympics without a tremendous investment, you would have to already have a lot of what the Olympics needs, going unused by your existing population. Friggin' Tokyo spent $2 billion in preparations. Tokyo. One of the largest, best-developed, best-organized megacities in the world wasn't ready to accommodate an Olympics.
I suppose some of a city's usual visitors would steer clear while there's an Olympics underway, freeing up things like hotel rooms and public transportation for people who are there specifically for the Olympics, so the infrastructure for accommodating visitors wouldn't need to be 100% in excess of its normal needs. Still, I think there's probably no city in the world that can accommodate an Olympics without building things it doesn't already use every day. Building things for an Olympics would have to coincide with some kind of boom immediately following the games, to take up the slack. Otherwise, you're just building a bunch of stuff that will be used for a handful of weeks and then abandoned. And I think much of what an Olympics needs that a city doesn't already have is of little use to that city under everyday circumstances, so the investment doesn't end up paying for itself. I think places like Montreal and Sochi got royally screwed. I suppose it remains to be seen whether the construction and investment can be absorbed by a metropolis the size of Tokyo (and we won't get an answer to that question this particular Summer, because of the pandemic - Tokyo/Japan is going to suffer a massive loss on its investment, at least partly due to the unusual circumstances).
Does anyone know what London invested in preparing for the 2012 Olympics, and whether they recouped it after the games were over?
Calgary turned a profit on the 1988 Winter Olympics, and some of the things built for that are
still being used. In fact, Calgary had a good chance of being chosen for the 2026 Winter Olympics, but given the situation with oil and gas and so many people struggling, many people who would otherwise have enjoyed the Olympics said no. I was appalled when I read that the province planned to kick in $700 million just for the bid, never mind what would have to spent to actually hold the event. Calgary held a plebiscite and voted not to hold the Games.
Those of us who were around in 1988 loved those Games. It was a very special time (I knew someone who performed in the Opening Ceremony), and it's what got me interested in figure skating and a whole slew of events I'd never even heard of. That was the year of Alberto Tomba, the Jamaican bobsled team, the Battle of the Brians, and Eddie "The Eagle" Edwards.
There was even a mini-Stampede, for the foreign visitors who wanted to see it but somehow thought that it runs all year, rather than two weeks in July. So we had a mini-Stampede in February, and when one of the CTV News anchors announced that one of the rodeo events held was ladies' barrel wrestling (

) ... people west of the Ontario/Manitoba border fell off their chairs laughing at that, but the anchor didn't even realize what a dumb thing he'd just said.
It was also fairly amusing when the Calgary organizers tried to convince the IOC and the American TV network that those particular weeks in February would really not be the best part of the winter in which to hold the games, due to the prevalence of the chinook winds (it can be freezing in the morning, the winds blow in over the mountains, and it's shirtsleeve weather by early afternoon, and this can continue for several days). They didn't listen, so it happened that they woke up one morning and wondered WTH happened to all the snow - the ski events had to be postponed because the snow had melted too much and they had to make artificial snow so the events could continue.
But thirty years is a long time, and Alberta just couldn't afford it. Especially when most of the province wouldn't get much benefit from it other than 2 weeks of sports on TV that we don't normally watch any other time.