A head-to-head comparison of Rotary Position Embedding and GPT-style learned position embeddings. Both 1.3B models were trained for 100k steps on the Pile using Mesh Transformer JAX. There isn't a very strong trend, but hopefully someone will find these results useful regardless.

Task Metric Learned Rotary
lambada ppl 7.940 ± 0.208 7.156 ± 0.208
acc 0.556 ± 0.007 0.567 ± 0.007
piqa acc 0.700 ± 0.011 0.714 ± 0.011
acc_norm 0.693 ± 0.011 0.709 ± 0.011
hellaswag acc 0.376 ± 0.005 0.389 ± 0.005
acc_norm 0.472 ± 0.005 0.488 ± 0.005
winogrande acc 0.540 ± 0.014 0.571 ± 0.014
mathqa acc 0.231 ± 0.008 0.230 ± 0.008
acc_norm 0.234 ± 0.008 0.227 ± 0.008
pubmedqa acc 0.599 ± 0.015 0.583 ± 0.015
boolq acc 0.575 ± 0.009 0.614 ± 0.009
anli_r3 acc 0.344 ± 0.014 0.351 ± 0.014
openbookqa acc 0.198 ± 0.018 0.206 ± 0.018
acc_norm 0.316 ± 0.021 0.330 ± 0.021
triviaqa acc 0.041 ± 0.002 0.026 ± 0.002
arc_challenge acc 0.235 ± 0.012 0.230 ± 0.012
acc_norm 0.260 ± 0.013 0.272 ± 0.013
arc_easy acc 0.564 ± 0.010 0.568 ± 0.010
acc_norm 0.505 ± 0.010 0.486 ± 0.010
cb acc 0.375 ± 0.065 0.357 ± 0.065
cola mcc 0.042 ± 0.034 0.022 ± 0.034
copa acc 0.730 ± 0.044 0.730 ± 0.044
ethics_cm acc 0.491 ± 0.008 0.480 ± 0.008
ethics_deontology acc 0.497 ± 0.008 0.497 ± 0.008
ethics_justice acc 0.501 ± 0.010 0.501 ± 0.010
ethics_utilitarianism acc 0.497 ± 0.007 0.493 ± 0.007
ethics_virtue acc 0.200 ± 0.006 0.200 ± 0.006
headqa acc 0.227 ± 0.008 0.224 ± 0.008
acc_norm 0.270 ± 0.008 0.271 ± 0.008
logiqa acc 0.221 ± 0.016 0.215 ± 0.016
acc_norm 0.293 ± 0.018 0.283 ± 0.018
mnli acc 0.344 ± 0.005 0.344 ± 0.005
mnli_mismatched acc 0.345 ± 0.005 0.349 ± 0.005
mrpc acc 0.684 ± 0.023 0.684 ± 0.023
f1 0.812 ± 0.017 0.812 ± 0.017
qa4mre_2011 acc 0.392 ± 0.045 0.358 ± 0.045
acc_norm 0.450 ± 0.045 0.433 ± 0.045
qa4mre_2012 acc 0.287 ± 0.036 0.312 ± 0.036
acc_norm 0.394 ± 0.039 0.400 ± 0.039
qa4mre_2013 acc 0.335 ± 0.028 0.335 ± 0.028
acc_norm 0.352 ± 0.028 0.349 ± 0.028
qnli acc 0.498 ± 0.007 0.517 ± 0.007
qqp acc 0.370 ± 0.002 0.368 ± 0.002
f1 0.538 ± 0.003 0.538 ± 0.003
race acc 0.345 ± 0.015 0.343 ± 0.015
record f1 0.805 ± 0.004 0.813 ± 0.004
em 0.797 ± 0.004 0.805 ± 0.004
rte acc 0.538 ± 0.030 0.523 ± 0.030
sciq acc 0.867 ± 0.011 0.865 ± 0.011
acc_norm 0.796 ± 0.013 0.771 ± 0.013
sst acc 0.572 ± 0.017 0.519 ± 0.017
webqs acc 0.021 ± 0.003 0.006 ± 0.003
wic acc 0.500 ± 0.020 0.498 ± 0.020
wnli acc 0.437 ± 0.059 0.549 ± 0.059
wsc acc 0.365 ± 0.047 0.365 ± 0.047
wsc273 acc 0.722 ± 0.027 0.736 ± 0.027