That's very interesting, as a Frenchman I assumed it came from "brutal", like "brutalité", as the raw concrete and abundance of right angles tends to make these buildings feel, well, aggressive (to me at least).

It is a tounge-in-cheek term derived by english speakers who were well aware of the double entendre. Sometimes even architects have a sense of humour.