'Opinion polls found that Leave voters believed leaving the EU was "more likely to bring about a better immigration system, improved border controls, a fairer welfare system, better quality of life, and the ability to control our own laws"'
Doesn't seem to have helped, has it?
So a justification based on a premise of alleviating some concerns about immigration has a long historical trail of failures behind it, as I'm sure the Windrush generation can share. The US and Canadian citizens along what was once pridefully called the world's longest unprotected border have also their misgivings.
As I read here, the UK passed the law that required employers to check employee eligibility. I'm sure that was meant to alleviate xenophobic concerns. Why wasn't that enough?
If anything, Brexit has exacerbated control over immigration as we can no longer access shared information with e.g. France.