Pretraining Data Filtering for Open-Weight AI Safety
Announcing Deep Ignorance: Filtering Pretraining Data Builds Tamper-Resistant Safeguards into Open-Weight LLMs
Research Notes
The Common Pile v0.1
Announcing the Common Pile v0.1: An 8TB Dataset of Public Domain and Openly Licensed Text
A Preliminary Exploration into Factored Cognition with Language Models
Exploring factored cognition by decomposing arithmetic tasks for GPT-3.
Multiple Choice Normalization in LM Evaluation
There are multiple ways of evaluating multiple-choice tasks on autoregressive language models like GPT-3/Neo/J. This post lays out the current prevalent normalization methods.
Downstream Evaluations of Rotary Position Embeddings
A comparison of Rotary Position Embedding against GPT-style learned position embeddings.
On the Sizes of OpenAI API Models
Using eval harness, we can deduce the sizes of OpenAI API models from their performance.
Evaluating Different Fewshot Description Prompts on GPT-3
We evaluate different fewshot prompts on GPT-3 to see how it changes performance.
Finetuning Models on Downstream Tasks
We tuned GPT-Neo on eval harness tasks to see how it would change its performance.
Activation Function Ablation
An ablation of activation functions in GPT-like autoregressive language models.