Or, using a proper lighting setup, so as to minimize shadows. This would likely solve most of their shadow-related problems.

You could also cluster first purely on colour (using the xy part of the xyY colourspace). For greyscale clothing you might then need to see if some colour has clearly separate shades, but that's going to be tricky without a proper understanding of shadows.

One could also remove the expected shadow values (and hues) depending on the color of the clothing and the color of the light, at least approximately.

Should work as long as there aren't complicated interactions of the material with light that hits it.