Part of that is just what we allow though. Too much red tape means only big builders that can get through it. Lot sizes and max units/lot means only expensive McMansions. Impact fees charged by municipal governments drive up prices and come directly out of the buyer's pocket (instead of using property taxes to find infrastructure). Never mind the general idea that new almost always costs more than used (though, in many areas, we're in such a rut, it would take tens to hundreds of thousands of new units across metro areas to really start making a dent in prices, either by slowing price growth, or of enough is built, actually lowering total prices)
At the end of the day, people need somewhere to live. If well off people are buying a home, they are likely leaving a place somewhere else that the next person in the economic ladder could occupy. Filtering is a real effect in housing.
Adjusting zoning and associated codes won't fix everything, but it is currently a severe constraint on what can be built in the first place. We also need targeted construction and subsidies to help out lower income people (but we shouldn't look to reproduce The Projects of the past).