Looking for a demo? Try GPT-NeoX-20B via CoreWeave and Anlatan’s inference service, GooseAI!
After a year-long odyssey through months of chip shortage-induced shipping delays, technical trials and tribulations, and aggressively boring debugging, we are happy to finally announce EleutherAI’s latest open-source language model: GPT-NeoX-20B, a 20 billion parameter model trained using our GPT-NeoX framework on GPUs generously provided by our friends at CoreWeave.
GPT-NeoX-20B is, to our knowledge, the largest publicly accessible pretrained general-purpose autoregressive language model, and we expect it to perform well on many tasks.
We hope that the increased accessibility of models of this size will aid in research towards the safe use of AI systems, and encourage anyone interested in working in this direction to reach out to us.
As a thank you to our generous compute donors, we are delaying the public downloadable release of the model by 7 days. On
There will be a #20b channel set up in our Discord for discussions of this model. Please note that much like our other language models and codebases, GPT-NeoX and GPT-NeoX-20B are very much research artifacts and we do not recommend deploying either in a production setting without careful consideration. In particular, we strongly encourage those looking to use GPT-NeoX-20B to read the paper and datasheet on our training data. There are still bugs to be ironed out and many inefficiencies that could be addressed—but hey, we do this in our free time, give us a break lol
|ANLI R1||Natural Language Inference||32.40%||32.80%||32.40%||34.00%||34.00%||36.30%|
|ANLI R2||Natural Language Inference||30.90%||33.50%||34.00%||33.00%||34.40%||37.00%|
|ANLI R3||Natural Language Inference||33.75%||35.50%||35.50%||34.75%||35.40%||36.83%|
Accuracy on standard language modeling tasks.
Zero-shot accuracy of factual knowledge by subject group, as measured by the HendrycksTest evaluation.