Could this be a hardware problem? I've been working on a WebGL project and often different graphics cards have different precision from 8 to 16 to 128 bits. I've seen similar quirks when precision truncates calculations.

Edit: Testing different hardware reminds me of the IE5-8 days. Sigh...