Firefox held off on AVIF for a few releases to get their image color spaces right. I'm usually blind when it comes to things like this (is this random image resized gamma correct? hell if I know), but even I could tell that AVIFs didn't look right before they fixed it. Those fixes were in well before FF 96, so there might be something else at play.

Tried on a different machine with FF 97 and the problem was gone.

So much stuff fucks with the color profile of the images, the browser, the OS, the screen. I wouldn't be surprised if only some of the test images have a color profile set correctly.