Under federal guidelines, households that spend more than 30 percent of their income on housing costs are considered "cost burdened" and will have difficulty affording basic necessities like food, clothing, transportation and medical care.
But the number of Americans struggling with their housing costs has risen from almost 16 million in 2001 to 38 million in 2015, according to the Census data crunched in the report. That's more than double.
And despite the overall economic recovery, it's only a small improvement from 2014, going down by about 900,000 households.
When people can't safely afford to pay their mortgages and rent, it isn't just a problem for those with a lower income or people who bit off more house than they can chew.
Let our news meet your inbox. The news and stories that matters, delivered weekday mornings.
Housing unaffordability also drags down GDP, slowing down overall economic growth for everyone, said Dan McCue, senior research associate at the Joint Center for Housing Studies at Harvard University, which publishes the annual State of the Nation's Housing report.
"It forces them to constrict spending on other items, which would reduce spending on other parts of the economy. They would buy less, save less, reduce savings," said McCue.
"It may make it more difficult to venture out and start a new company — or, living month to month, they're much less likely to go back to school and get additional training; and may not be in the job that makes them the most productive member of the labor market," McCue told NBC News.
A big factor has been how wages haven't kept pace with rising housing costs.
"For lower income groups, it's even worse than stagnation. It's not keeping up with inflation," said McCue.
Housing costs are being driven by a limited supply of move-in quality, entry-level housing, said Diane Swonk, CEO of DS Economics.
"In the wake of the financial crisis, so much capacity was taken offline," Swonk told NBC News. "Much of the existing stock of housing is still underwater. Many of the entry level houses are in disrepair."
And what building is happening is happening upmarket.
"Builders are less able to downscale and build smaller volumes of smaller homes," said Swonk. "It's restricting supply well below demand, so of course it shows up in price."
Also factoring in is a net decline in migration from Mexico after 2009 that decreased the number of skilled construction workers, and an increase in material costs.
Ben Popken is a senior business reporter for NBC News.