IANAG but the idea is that women get two copies of the X chromosome (XX), and men only one (XY). This explains why women have squared the colour blindness rates of men - women have to get two bad copies, men only one.

Many intelligence related genes are on the X chromosome, so it makes sense you get more variation in men. However, not all genes interact in this way.