You can manually program transformers:

https://srush.github.io/raspy/

I don't know if you can integrate them into a model. I think you might run out of space, since these aren't polysemantic and so would take up a lot more "room" than learned neurons.