You'd obviously not be buying the same house... you'd be paying for a $100k house. You're saying it would be less nice, basically. I say, banks want you to think you need the bigger house. You don't.
How expensive of a house you want or need depends on the person and location. And you are now saving money by buying less, of course you are, this discussion only exsists if you are talking about equivalent houses. I can make the same mortgage arguments about your smaller house.