What We Learned Trying to Catch AI Liars: An Aletheia's Quest Retrospective
What we learned while building black-box and white-box detectors for AI deception during Aletheia's Quest.
Read post →Research Notes, Announcements, and Technical Essays
Latest Posts
FineBooks: are open OCR models good enough to unlock historical knowledge?
Testing open OCR models on historical books with the FineBooks BHL OCR Leaderboard.
Read post →A Dynamical Model of AI Governability
A toy dynamical model of whether the AI workforce that builds future AI ends up cooperative or uncooperative: where the basin boundary lies, what current evidence says about which side we are on, and what would tell us we are on the good path.
Read post →Recent Archive
Early Indicators of Reward Hacking via Reasoning Interpolation
Using importance sampling with fine-tuned donor prefills to predict reward hacking emergence during training
Reward Hacking Research Update
Interim report on ongoing work on reward hacking
Pretraining Data Filtering for Open-Weight AI Safety
Announcing Deep Ignorance: Filtering Pretraining Data Builds Tamper-Resistant Safeguards into Open-Weight LLMs
Attention Probes
Adding attention to linear probes
Research Update: Applications of Local Volume Measurement
Research update on applying local volume measurement to downstream tasks
Studying inductive biases of random networks via local volumes
In this post, we will study inductive biases of the parameter-function map of random neural networks using star domain volume estimates. This builds on the ideas introduced in Estimating the Probability of Sampling a …
The Common Pile v0.1
Announcing the Common Pile v0.1: An 8TB Dataset of Public Domain and Openly Licensed Text
Product Key Memory Sparse Coders
Using Product Key Memories to encode sparse coder features
SAEs trained on the same data don’t learn the same features
Comparing feature overlap and interpretability across TopK sparse autoencoders trained with different random seeds.
Partially rewriting an LLM in natural language
Using interpretations of SAE latents to simulate activations.
Third-party evaluation to identify risks in LLMs’ training data
A demonstration of third-party evaluation of private language-model training data using PySyft.
Mechanistic Anomaly Detection Research Update 2
Interim report on ongoing work on mechanistic anomaly detection
RLHF and RLAIF in GPT-NeoX
GPT-NeoX now supports post-training thanks to a collaboration with SynthLabs.
The Practitioner's Guide to the Maximal Update Parameterization
Exploring the implementation details of muTransfer
Mechanistic Anomaly Detection Research Update
Interim report on ongoing work on mechanistic anomaly detection