Tele-FLM

Model overview

Tele-FLM (aka FLM-2) is a 52B open-sourced multilingual large language model that features a stable, efficient pre-training paradigm and enhanced factual judgement capabilities. Built upon the decoder-only transformer architecture, it has been trained on approximately 2T tokens. Tele-FLM demonstrates superior performances at its scale, and sometimes surpass larger models. In addition to sharing the model weights, we provide the core designs, engineering practices, and training details, anticipating their benefits for both academic and industrial communities. Although we've made extensive efforts to thoroughly clean and filter the training corpus for the model, due to the open nature of the dataset, the model may still have picked up on some unsafe examples. Consequently, the model may still generate unexpected content, including but not limited to discrimination, bias, or offensive language. We would like to strongly advise users not to spread any unsafe content generated by the model. The project developers cannot be held responsible for any repercussions stemming from the dissemination of harmful information. Our training dataset comprises a variety of domains, as detailed in the table below. The total amount of data is roughly 2 trillion, with English and Chinese data in a ratio of about 2:1. In line with the methodology of GPT-4, we collected some instruct data and incorporated it into our pre-training data after removing the test sets of common datasets using the strict n-gram-based method. We deliberately avoid “training on the test set” or any other benchmark-oriented trick. |Domain |Language|Sampling Prop. |Epochs |Disk Size | |-------|:--------------:|:--------------:|:-------:|:-----------:| | Webtext |en, zh | 75.21% | 1.0 | 5.9 TB | | Code |code, zh | 9.81% | 1.0 | 528.1 GB | | Book |en, zh | 7.17% | 0.8 | 647.6 GB | | WorldKnowledge |multi, en, zh | 2.87% | 2.5 | 67.5 GB | | QA |en, zh | 2.12% | 1.0 | 159.2 GB | | AcademicPaper |en | 0.99% | 1.0 | 54.4 GB | | Profession-Law |zh | 1.04% | 1.0 | 84.2 GB | | Profession-Math |math | 0.62% | 2.0 | 6.1 GB | | Profession-Patent |zh | 0.14% | 1.0 | 10.4 GB | | Profession-Medical |zh | 0.02% | 1.0 | 1.2 GB |

Key features

特点摘要待依据来源补齐，不凭模型名称推断能力。

Limitations & considerations

Sources are insufficient to establish specific limitations. Verify task performance, latency, data policies, licensing and version lifecycle.

Source prices & cost analysis

Prices vary by channel, region and configuration. Channel quotes are listed independently and are not treated as a single official provider price.

Public benchmarks & run configuration

暂无已采集的可核验公开评分。模型未上榜不代表能力较低。

Frequently asked questions

Sources & traceability

许可证：apache-2.0。模型介绍与第三方评测均以原来源的具体版本和方法为准。