Blog

Discussing and disseminating open-source AI research.

2025

June

Studying inductive biases of random networks via local volumes

June 12, 2025 · Louis Jaburi, Nora Belrose

The Common Pile v0.1

June 5, 2025 · Stella Biderman, Sebastian Majstorovic, Aviya Skowron

May

Product Key Memory Sparse Coders

May 30, 2025 · Stepan Shabalin, Gonçalo Paulo, Nora Belrose

2024

December

SAEs trained on the same data don’t learn the same features

December 12, 2024 · Gonçalo Paulo, Nora Belrose

November

Partially rewriting an LLM in natural language

November 10, 2024 · Gonçalo Paulo, Nora Belrose

October

Third-party evaluation to identify risks in LLMs’ training data

October 31, 2024 · Irina Bejan, Curtis Huebner

Mechanistic Anomaly Detection Research Update 2

October 14, 2024 · David Johnston, Nora Belrose

RLHF and RLAIF in GPT-NeoX

October 10, 2024 · Dakota Mahan, Quentin Anthony, Louis Castricato, Nathan Lile, Stella Biderman

September

The Practitioner's Guide to the Maximal Update Parameterization

September 19, 2024 · Nolan Dey, Quentin Anthony, Joel Hestness

August

Mechanistic Anomaly Detection Research Update

August 5, 2024 · David Johnston, Arkajyoti Chakraborty, Nora Belrose

July

Open Source Automated Interpretability for Sparse Autoencoder Features

July 30, 2024 · Caden Juang, Gonçalo Paulo, Jacob Drori, Nora Belrose

June

Experiments in Weak-to-Strong Generalization

June 14, 2024 · Adam Scherlis, Alex Mallen, Lucia Quirke, Nora Belrose

Free Form Least-Squares Concept Erasure Without Oracle Concept Labels

June 13, 2024 · Brennan Dury, Nora Belrose

May

VINC-S: Closed-form Optionally-supervised Knowledge Elicitation with Paraphrase Invariance

May 23, 2024 · Nora Belrose*, Alex Mallen*, Dhruba Ghosh, Walter Laurito, Kyle O'Brien, Alexander Wan, Ben Wright, Akari Asai, Yanai Elazar

April

Pile-T5

April 15, 2024 · Lintang Sutawika, Aran Komatsuzaki, Colin Raffel

March

Yi-34B, Llama 2, and common practices in LLM training: a fact check of the New York Times

March 25, 2024 · Hailey Schoelkopf, Aviya Skowron, Stella Biderman

February

The Foundation Model Development Cheatsheet

February 29, 2024 · Hailey Schoelkopf, Stella Biderman

2023

December

Least-Squares Concept Erasure with Oracle Concept Labels

December 19, 2023 · Nora Belrose

Diff-in-Means Concept Editing is Worst-Case Optimal

December 11, 2023 · Nora Belrose

November

The third New England RLHF Hackers Hackathon

November 26, 2023 · Will Beddow, Matthew Bernstein, Chase Blagden, Louis Castricato, Jacob Makar-Limanov, Arjun Prakash, Sid Verma

Extending the RoPE

November 13, 2023 · Honglu Fan, Bowen Peng, Jeffrey Quesnelle

October

How the Foundation Model Transparency Index Distorts Transparency

October 26, 2023 · Nathan Lambert, SE Gyges, Stella Biderman, Aviya Skowron

Llemma: An Open Language Model For Mathematics

October 16, 2023 · Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer', Albert Q. Jiang, Jia Deng, Stella Biderman, Sean Welleck

The second New England RLHF Hackers Hackathon

October 13, 2023 · Suraj Anand, Chase Blagden, Louis Castricato, Yeganeh Kordi, Francisco Piedrahíta-Vélez, Arjun Prakash, Logan Riggs, Benjamin Spiegel, Kevin A. Wang, Benjamin Wright, Yong Zheng-Xin, Max Zuo

September

Contributor Spotlight: Mohammad Aflah Khan

September 21, 2023 · Mohammad Aflah Khan

The first New England RLHF Hackers Hackathon

September 19, 2023 · Suraj Anand, Stephen Casper, Louis Castricato, Arjun Khurana, Alyssa Loo, Shukai Ni, Benjamin Spiegel, Jiayu Zheng, Yong Zheng-Xin

July

EleutherAI's Thoughts on the EU AI Act

July 26, 2023 · Aviya Skowron, Stella Biderman

Minetester: A fully open RL environment built on Minetest

July 8, 2023 · Curtis Huebner, Robert Klassert, Stepan Shabalin, Edwin Fennell, Delta Hessler

May

🐶Safetensors audited as really safe and becoming the default

May 23, 2023 · Nicolas Patry, Stella Biderman, Garry Jean-Baptiste

Alignment Research @ EleutherAI

May 3, 2023 · Curtis Huebner

April

Transformer Math 101

April 18, 2023 · Quentin Anthony, Stella Biderman, Hailey Schoelkopf

Exploratory Analysis of TRLX RLHF Transformers with TransformerLens

April 2, 2023 · Curt Tigges

March

EleutherAI Second Retrospective: The long version

March 26, 2023 · Stella Biderman, Shivanshu Purohit, Curtis Huebner, Leo Gao, Connor Leahy, Eric Hallahan

The View from 30,000 Feet: Preface to the Second EleutherAI Retrospective

March 2, 2023 · Stella Biderman, Curtis Huebner, Connor Leahy, Eric Hallahan

2022

February

Announcing GPT-NeoX-20B

February 2, 2022 · Connor Leahy

2021

October

A Preliminary Exploration into Factored Cognition with Language Models

October 25, 2021 · Leo Gao, Kyle McDonell, Laria Reynolds, Stella Biderman

Multiple Choice Normalization in LM Evaluation

October 11, 2021 · Leo Gao

August

Downstream Evaluations of Rotary Position Embeddings

August 16, 2021 · Leo Gao

July

What A Long, Strange Trip It's Been: EleutherAI One Year Retrospective

July 7, 2021 · Connor Leahy, Eric Hallahan, Leo Gao, Stella Biderman

June

Why Release a Large Language Model?

June 2, 2021 · Connor Leahy

May

On the Sizes of OpenAI API Models

May 24, 2021 · Leo Gao

Evaluating Different Fewshot Description Prompts on GPT-3

May 24, 2021 · Leo Gao

Finetuning Models on Downstream Tasks

May 24, 2021 · Leo Gao

Activation Function Ablation

May 24, 2021 · Leo Gao

April

Rotary Embeddings: A Relative Revolution

April 20, 2021 · Stella Biderman, Sid Black, Charles Foster, Leo Gao, Eric Hallahan, Horace He, Ben Wang, Phil Wang