https://blog.eleuther.ai/categories/investigations/ Recent content in Investigations on EleutherAI Blog Hugo -- gohugo.io en-us Tue, 18 Apr 2023 00:00:00 +0100 https://blog.eleuther.ai/transformer-math/ Tue, 18 Apr 2023 00:00:00 +0100 https://blog.eleuther.ai/transformer-math/ We present basic math related to computation and memory usage for transformers https://blog.eleuther.ai/rotary-embeddings/ Tue, 20 Apr 2021 21:00:00 -0400 https://blog.eleuther.ai/rotary-embeddings/ Rotary Positional Embedding (RoPE) is a new type of position encoding that unifies absolute and relative approaches. We put it to the test.