Would you ever use ReLU in a new architecture? When?

For standard pre-training with a modern stack, there is no compelling reason to choose ReLU over SwiGLU. However, it may be used if there is a specific need for sparsity-based inference acceleration via squared ReLU, or if the deployment environment is highly constrained and lacks the necessary kernels for more advanced activation functions.