Small models, big results: Achieving superior intent extraction through decomposition
Generative AI
AI NEWS JOURNAL
Search AI news
Search headlines, summaries and articles to connect the dots.
NeuralGCM harnesses AI to better simulate long-range global precipitation
Climate & Sustainability
Reducing EV range anxiety: How a simple AI model predicts port availability
Algorithms & Theory
We’re expanding our presence in Singapore to advance AI in the Asia-Pacific region
Google DeepMind opens a new Singapore research lab, accelerating AI progress in the Asia-Pacific region.
Separating natural forests from other tree cover with AI for deforestation-free supply chains
Climate & Sustainability
What exactly does word2vec learn?
What exactly does word2vec learn, and how? Answering this question amounts to understanding representation learning in a minimal yet interesting language modeling task. Despite the fact that word2vec is a well-known precursor to modern language models, for many years, researchers lacked a quantitative and predictive theory describing its learning process. In our new paper, we finally provide such a theory. We prove that there are realistic, practical regimes in which the learning problem reduces to unweighted least-squares matrix factorization. We solve the gradient flow dynamics in closed form; the final learned representations are simply given by PCA. Learning dynamics of word2vec. When trained from small initialization, word2vec learns in discrete, sequential steps. Left: rank-increment