As a vague platitude yes, total output has to be reduced by some amount if labor input is reduced. When actually thinking of concrete impacts on society, though, the quantity and quality of many kinds of societal outputs do not scale linearly with overall work-hours, nevermind that relative allocation of work-hours across different fields and disciplines would be scaled in nonlinear ways (see also: complaints about make-work and "bullshit jobs"). And housing in particular is almost universally bottlenecked by the supply of land and limitations in organization/planning of cities and towns rather than any shortage of construction labor.
Given all that, "houses would be proportionally fewer, smaller, and and in need for repair...unless we assume builders and maintenance people are exempt from working less" is not actually self-evident, and is more likely to be taken as an attempt to paint an exaggerated picture for rhetorical purposes.
I think it is a less exaggerated take than the idea we can all work half as much and have the same amount of stuff.
There are lots of things that can be done, but they revolve around increasing either productivity or increasing efficiency. Neither of these is synonymous with simply cutting back on work.
He's saying that, in the places where people do work harder, they don't actually have more/larger/better houses. Rather, the work just becomes part of a zero-sum competition that bids up the price of the (fixed) quantity/quality of houses.
This partially contradicts your point.
What I would add (to reconcile the two points), is that one kind of work is not fungible with another kind of work. Yes, people work very hard in Silicon Valley -- but they are not working hard at building houses. If they were, there'd be a lot of supply, and the price would fall.
Overall, this is perhaps a comment about the (mis-)allocation of work in society.
Working less might give people more time to "sharpen the axe", both on an individual level and as a society.
A large portion of workers believe their own job is "bullshit" and does nothing to benefit society. Perhaps if we had more breathing room we could find ways to move workers into meaningful jobs.
It's complicated, and working fewer hours doesn't necessarily mean less productivity.
My point is simply that if everyone works less, society will have proportionally less material stuff.