Also it is wrong. The paper has add and norm, but the official implementation and all other good implementation has pre norm architecture: https://twitter.com/francoisfleuret/status/14671353665032192...