When people want a smooth function so they can do calculus they often use someth...

		seanhunter 7 days ago \| parent \| context \| favorite \| on: Failing to Understand the Exponential, Again When people want a smooth function so they can do calculus they often use something like gelu or the swish function rather than relu. And the swish function involves a sigmoid. https://en.wikipedia.org/wiki/Swish_function

The gated variants of these functions have been dominant for a few years.