The EleutherAI Research Notes are an experiment in sharing bites of preliminary research results through short and informal posts rather than the occasional paper. Our hope for this format is to encourage low-friction sharing of ideas in the fast-paced world of modern research.
Research Notes
The second New England RLHF Hackers Hackathon
The first New England RLHF Hackers Hackathon
A Preliminary Exploration into Factored Cognition with Language Models
Multiple Choice Normalization in LM Evaluation
There are multiple ways of evaluating multiple choice tasks on autoregressive LMs like GPT-3/Neo/J. This post lays out the current prevalent normalization methods.
Downstream Evaluations of Rotary Position Embeddings
A comparison of Rotary Position Embedding against GPT-style learned position embeddings.
On the Sizes of OpenAI API Models
Using eval harness, we can deduce the sizes of OpenAI API models from their performance.
Evaluating Different Fewshot Description Prompts on GPT-3
We evaluate different fewshot prompts on GPT-3 to see how it changes performance.
Finetuning Models on Downstream Tasks
We tuned GPT-Neo on eval harness tasks to see how it would change its performance.
Activation Function Ablation
An ablation of activation functions in GPT-like autoregressive language models.