From what I read on the topic, it's much more complicated. People have plenty of typical-female or typical-male parts of brain slushed around, generally people tend to have the most of them that correspond to their gender (IIRC you can guess gender/sex correctly about 80 or 90% of the time by looking at the brain).
It may contribute along with other factors for the cases where the brain does express gender dysphoria.
Gender dysphoria also isn't a clean distinction between "male" and "wanting to be female". The underlying phenomenon is observed to be rooted in sex-specific instincts and feelings inverse to what is normally found in people of the same sex. The interpretation is cultural. Indians believe it means there are two spirits in a single body, Mexicans have the "mux", and even in the west there's been tacit admission by some trans people that they feel different but couldn't tell you they were actually female...the oft-maligned "nonbinary".
So really, I don't personally believe a brain can be cleanly male or cleanly female. Rather, if specific parts are formed differently than usual, then they respond differently to external stimuli. This can result in a wide range of sex-atypical behaviour ranging from mere homosexuality to finding one's genitals revolting. How you interpret that objective phenomenon is really up to the individual.
I think the brains that are cleanly male-trait or female-trait are very very rare, most are a mix but there is usually enough difference to be able to make a very good guess.
Though I also do agree that gender dysphoria can have many different expressions, once you drop gender you get into a whole new world of stuffs where people hate various things about themselves (I've met people with self-described "whole body dysphoria" who are generally okay with their gender and simply hate the physical existence of their body)
To grossly summarize a cocktail factoid - bird cells can individually differentiate, and some birds are truly half and half. Us mammals bombard our cells with hormones, constantly telling them how to differentiate, as though we were some sort of eusocial insect colony.
Yes, other animals have found rather interesting ways to differentiate into genders, IIRC in some cases it's temperature, in others hormones, others do it genetically and I think in a few it's merely "how many females are in my group?", where gender can flip to benefit reproduction.
