IT Home reported on October 30 that users of less widely spoken languages may face higher costs when using AI services compared to users of English and other major languages.
According to a report by foreign media outlet Rest of World on the 28th local time, a study released last month by the Artificial Intelligence Language Equity Research Consortium found that AI models can consume vastly different numbers of tokens when processing various languages.
Tokens are the fundamental units used by AI models to process text and other data, generated by tokenizers that break down text, images, or audio into smaller components. Researchers refer to the additional costs arising from these differences as an invisible "tokenizer tax," with languages having limited online content being particularly vulnerable.
The study revealed that even among the 20 most commonly used languages on the internet, there are significant variations in token consumption—ranging from 1.5 to 5 times between the highest and lowest values. Expanding the scope to 200 languages would further widen this gap. In one particular AI model, the difference in token consumption across languages reached as much as 17-fold.
Author’s Note: This image was generated by AI
The research report points out that a startup based in Bangalore might incur higher costs each time it inputs instructions into an AI system and receives responses, compared to its competitors in Austin. The issue caused by the "tokenizer tax" goes beyond just increased expenses: the more tokens consumed, the slower the AI response becomes, and the less contextual information it can effectively process.
Regions already constrained by limited computing resources and network conditions are especially hard hit. This inequality in tokenization also restricts people's opportunities to participate in the AI economy.
Pratyush Chaudhary, co-founder of Activate AI, India’s first venture capital fund focused on AI investments, points out that when working with non-English languages, the same contextual capacity can accommodate fewer documents and conversation records, making it more challenging for AI to generate responses.
Using education as an example, Chaudhary explains: “For a teacher to effectively instruct students, they must remember past mistakes and previous explanations. If, when using a particular language, these historical records quickly fill up the context window, even teaching quality will ultimately suffer.”
Chaudhary believes that language-processing efficiency should be considered a fundamental requirement for AI products, alongside accuracy, speed, and reliability. “It may not be realistic to demand complete equality across all languages, but the long-standing significant disparities still require clear explanations and concrete improvement plans. I hope all major AI labs will publish evaluation results for each language, disclose how much it costs to accomplish tasks of similar difficulty in different languages, and commit to improving support for the weakest languages, achieving quantifiable progress.”
IT Home learned from reports that one way to address this issue is by enhancing the quality of underlying data.
Raul Nanavati, co-founder and CEO of Navana.ai, an AI voice infrastructure startup based in Mumbai, notes: “Open-data initiatives are crucial in this regard. For instance, our RESPIN project—conducted in collaboration with the Indian Institute of Science and funded by the Gates Foundation—has collected text and audio recordings from native speakers of 38 dialects.”
However, the Artificial Intelligence Language Equity Research Consortium has found that token consumption isn’t determined solely by the number of speakers of a given language; the writing system used also plays a critical role.
For example, Asturian has relatively few speakers and is not included in the Common Crawl dataset, yet its use of the Latin alphabet gives it an advantage. Similarly, other Romance-language varieties like Italian are represented in the dataset, which also benefits Asturian.
Dheemanth Reddy, CEO of Maya Research, points out that many existing tokenizers remain primarily optimized for data-rich languages such as English. The root problem lies in the lack of high-quality digitized data.
Reddy emphasized that improving the training data alone is not enough: “Token prices are dropping rapidly, so the direct impact of cost disparities should gradually diminish. Higher-quality multilingual data and synthetic data can also help enhance model performance. However, to truly narrow the tokenization gap between different languages, AI labs still need to specifically refine the design of next-generation tokenizers, enabling them to handle these languages and scripts more effectively.”