I could chunk of the brain is visual processing. If people with history of mild infections have brain damage, wouldn't they commonly have visual disturbances? I've not heard any reports of anything of the sort.
Even being vaccinated doesn’t completely protect you from the cognitive effects of Covid. Everyone who’s had it has some level of permanent brain damage.
Yes, of course, nothing is 100%. But vaccines reduce both the severity and duration of illness, thereby reducing the risk and severity of brain damage.