AIが動く。未来が更新される。reformer-enwik8
*enwik8* is a dataset based on Wikipedia and is often used to measure the model's ability to *compress* data, *e.g.* in the scope of the *Hutter prize*: https://en.wikipedia.org/wiki/Hutter_Prize. `reformer-enwik8` was pretrained on the first 90M chars of *enwik8* whereas the text was chunked into batches of size 65536 chars (=2^16). The model's weights were taken from https://console.cloud.google.com/storage/browser/trax-ml/reformer/enwik8 and converted to Hugging Face's PyTorch ReformerLM model `ReformerModelWithLMHead`. The model is a language model that operates on characters. Therefore, this model does not need a tokenizer. The following function can instead be used for **encoding** and **decoding**:
翻訳準備中のため原文を表示しています
*enwik8* is a dataset based on Wikipedia and is often used to measure the model's ability to *compress* data, *e.g.* in the scope of the *Hutter prize*: https://en.wikipedia.org/wiki/Hutter_Prize. `reformer-enwik8` was pretrained on the first 90M chars of *enwik8* whereas the text was chunked into batches of size 65536 chars (=2^16). The model's weights were taken from https://console.cloud.google.com/storage/browser/trax-ml/reformer/enwik8 and converted to Hugging Face's PyTorch ReformerLM model `ReformerModelWithLMHead`. The model is a language model that operates on characters. Therefore, this model does not need a tokenizer. The following function can instead be used for **encoding** and **decoding**: