The EleutherAI Research Notes are an experiment in sharing bites of preliminary research results through short and informal posts rather than the occasional paper. Our hope for this format is to encourage low-friction sharing of ideas in the fast-paced world of modern research.
Announcing Deep Ignorance: Filtering Pretraining Data Builds Tamper-Resistant Safeguards into Open-Weight LLMs
Announcing the Common Pile v0.1: An 8TB Dataset of Public Domain and Openly Licensed Text
There are multiple ways of evaluating multiple choice tasks on autoregressive LMs like GPT-3/Neo/J. This post lays out the current prevalent normalization methods.
A comparison of Rotary Position Embedding against GPT-style learned position embeddings.
Using eval harness, we can deduce the sizes of OpenAI API models from their performance.
We evaluate different fewshot prompts on GPT-3 to see how it changes performance.
