santacoder

The SantaCoder models are a series of 1.1B parameter models trained on the Python, Java, and JavaScript subset of The Stack (v1.1) (which excluded opt-out requests). The main model uses Multi Query Attention, a context window of 2048 tokens, and was trained using near-deduplication and comment-to-code ratio as filtering criteria and using the Fill-in-the-Middle objective. In addition there are several models that were trained on datasets with different filter parameters and with architecture and objective variations. The `final` model is the best performing model and was trained twice as long (236B tokens) as the others. This checkpoint is the default model and available on the `main` branch. All other checkpoints are on separate branches with according names. The model was trained on GitHub code. As such it is _not_ an instruction model and commands like "Write a function that computes the square root." do not work well. You should phrase commands like they occur in source code such as comments (e.g. `# the following function computes the sqrt`) or write a function signature and docstring and let the model complete the function body.

译文尚未就绪，显示来源原文