This was an ablation of activation functions on GPT-like models of ~100M params that I ran ages ago. Each model was run for 10k iters, which isn't very long. My original goal was to show that activation function doesn't matter than much, but to do so I'd need to run a bunch more runs to get variance and show no statistical significance, and I don't plan on running a more exhaustive version of this experiment any time soon. So, I'm just dumping these results here in case anyone has any use for them. All the activation definitions are here.
| Name | Pile Validation BPB | LAMBADA acc | LAMBADA ppl |
|---|---|---|---|
| softsign | 1.1485 | 34.3 | 81.32 |
| ReLU | 1.1482 | 34.3 | 82.01 |
| spike2 | 1.1480 | 34.4 | 83.13 |
| selu | 1.1485 | 34.5 | 83.32 |
| elish | 1.1492 | 33.9 | 84.04 |
| tanhexp | 1.1474 | 33.7 | 84.06 |
| sigmoid | 1.1484 | 33.9 | 85.20 |
| tanhshrink | 1.1483 | 33.9 | 85.42 |
| maxtanh | 1.1479 | 33.7 | 85.53 |
| roottanh | 1.1485 | 33.4 | 86.00 |
| softplusmone | 1.1488 | 34.1 | 86.21 |
| logsoftmax | 1.1492 | 34.2 | 86.29 |
| ELU | 1.1496 | 33.8 | 86.37 |
| Swish | 1.1482 | 33.7 | 86.42 |
| softmax | 1.1491 | 33.2 | 86.74 |
| square_relax | 1.1484 | 33.5 | 86.92 |
| lisht | 1.1500 | 33.8 | 87.17 |
| GELU | 1.1453 | 34.0 | 87.84 |
| abs | 1.1489 | 33.5 | 87.96 |
| tanh | 1.1481 | 33.2 | 89.28 |
| Mish | 1.1482 | 33.6 | 89.84 |
| triangle_relax | 1.1502 | 33.7 | 89.91 |
| seagull | 1.1487 | 33.3 | 90.08 |
| maxsig | 1.1480 | 33.3 | 90.23 |
| softplus | 1.1460 | 33.1 | 90.74 |
| minsin | 1.1498 | 33.3 | 91.18 |
| snake | 1.1484 | 33.1 | 91.93 |
| cosid | 1.1490 | 33.3 | 92.99 |
| spike | 1.1498 | 33.3 | 93.78 |
| bipolarsigmoid | 1.1513 | 32.8 | 96.73 |