AI is happening. The future is updating.pythia-14m-deduped
> **Note:** Pythia-14M-deduped was trained after the original Pythia suite at the request of interpretability researchers who wanted a smaller model with the same training setup. It uses the same data, tokenizer, hyperparameter conventions, and checkpoint schedule as the rest of the Pythia suite. Unlike the original Pythia models (70M--12B), Pythia-14M-deduped was only trained on the deduplicated Pile and does not have a separate non-deduplicated variant. The primary intended use of Pythia is research on the behavior, functionality, and limitations of large language models. This suite is intended to provide a controlled setting for performing scientific experiments. We provide 154 checkpoints per model: - Initial `step0` - 10 log-spaced checkpoints `step{1,2,4...512}` - 143 evenly-spaced checkpoints from `step1000` to `step143000` These checkpoints are hosted on Hugging Face as branches. Note that branch `143000` corresponds exactly to the model checkpoint on the `main` branch of each model.
> **Note:** Pythia-14M-deduped was trained after the original Pythia suite at the request of interpretability researchers who wanted a smaller model with the same training setup. It uses the same data, tokenizer, hyperparameter conventions, and checkpoint schedule as the rest of the Pythia suite. Unlike the original Pythia models (70M--12B), Pythia-14M-deduped was only trained on the deduplicated Pile and does not have a separate non-deduplicated variant. The primary intended use of Pythia is research on the behavior, functionality, and limitations of large language models. This suite is intended to provide a controlled setting for performing scientific experiments. We provide 154 checkpoints per model: - Initial `step0` - 10 log-spaced checkpoints `step{1,2,4...512}` - 143 evenly-spaced checkpoints from `step1000` to `step143000` These checkpoints are hosted on Hugging Face as branches. Note that branch `143000` corresponds exactly to the model checkpoint on the `main` branch of each model.