AI 与机器学习jalammar.github.io编辑精选
图解 Word2vec
The Illustrated Word2vec
2019 — 2024，它一次次回到首页
黑点是上首页（≥ 50 分）的那次投稿，空心点是没冲上去的投稿，点越大分数越高。
导读
这是一篇图解教程，用可视化方式讲解词向量（word2vec）的核心原理，重点拆解了带负采样的 skip-gram（SGNS）算法的工作机制。文章还延伸到向量表示在推荐系统等场景中的应用，帮助读者直观理解“用向量表示词语”的价值。
为什么经久不衰：它把抽象的词嵌入算法讲得直观易懂，是 NLP 入门常被推荐的经典图解，多年后仍是理解嵌入概念的绝佳起点。
HN 评论精选
纠正误解：word2vec 的单个维度并不对应“王室”这类具体概念，维度含义只是“旋转等价”的。
认为“king”和“queen”在某些维度上相似、可能编码了“王室”概念，这是一个常见误解，而“人格嵌入”的例子反而加深了它。直觉上很容易理解为什么通常不是这样：旋转向量空间会保持所有词之间的余弦相似度不变。假设第 20 个分量编码了“王室”，那么存在无穷多种旋转方式会把“王室”分散到许多维度上。比如用 0-1 的值表示人格向量的开放性-外向性，两个人 [0,1] 和 [1,0] 是正交的，余弦相似度为 0；把空间旋转 45 度后变成 [-0.707,0.707] 和 [0.707,0.707]，余弦相似度仍然是 0，但人格特质到分量的直接映射显然丢失了。word2vec 的训练目标中没有任何东西偏好“维度有可解释含义”的向量空间。
danieldk查看英文原文
There are clear places where “king” and “queen” are similar to each other and distinct from all the others. Could these be coding for a vague concept of royalty? This is a common misunderstanding and unfortunately strengthened by the example of 'personality embeddings'. It is easy to understand intuitively why this is normally not the case. If you rotate a vector/embedding space, all the cosine similarities between words are preserved. Suppose that component 20 encoded 'royalty', there is an infinite number of rotations of the vector space where 'royalty' is distributed among many dimensions. Consider e.g. the personality vector openness-extroversion as values between 0-1. Now we have two persons: [0, 1] [1, 0] These vectors are orthogonal, so the cosine similarity is 0. Now let's rotate the vector space by 45 degrees: [-0.707, 0.707] [0.707, 0.707] The cosine similarity between the two vectors is still 0. Clearly, the direct mapping of personality traits to vector components is lost (as can be seen in the second vector). Obviously, this is something that you generally do not want to do for personality vectors. However, there is nothing in the word2vec objectives that would prefer a vector space with meaningful dimensions. E.g. take the skip-gram model, which maximizes the log-likelihood of the probability of a context word f, cooccuring with a context word c. Shortened: p(1|w_f,w_c) = 𝜎(w_f·w_c). So, the objective in vanilla word2vec prefers vector spaces that maximize the inner product of words that co-occur and minimize the inner product of words that do not co-occur. Consequently, if we have an optimal parametrization of W and C (the word and context matrices), any rotation of the vector space is also an optimal solution. Which rotation you actually get is dependent on accidental factors, such as the initial (typically randomized) parameters. Of course, it is possible to rotate the vector space such that dimensions become meaningful (see e.g. [1]), but with word2vec's default objective meaningful dimensions are purely accidental, and the meaning of vectors is defined in their relation to other vectors / their neighborhood. [1] https://www.aclweb.org/anthology/D17-1041
作者说明写作动机：视觉化讲解 SGNS，并为推荐系统使用者提供入门。
作者自述：写这篇文章是想用视觉化方式解释 word2vec 的 skip-gram + 负采样（SGNS）算法。动机一是为嵌入算法建立更多视觉化的表达语言，二是给把 SGNS 用于推荐系统的人提供一个平缓的入门路径（这个用例很有意思，文中有相关链接）。也希望它能帮助向新手解释向量表示的价值。
jalammar查看英文原文
Hello HN, Author here. I wrote this blog post attempting to visually explain the mechanics of word2vec's skipgram with negative sampling algorithm (SGNS). It's motivated by: 1- The need to develop more visual language around embedding algorithms. 2- The need for a gentle on-ramp to SGNS for people who are using it for recommender systems. A use-case I find very interesting (there are links in the post to such applications) I'm hoping it could also be useful if you wanted to explain to someone new to the field the value of vector representations of things. Hope you enjoy it. All feedback is appreciated!
强调维度无固有含义，并推荐一个更清晰的 PyTorch 实现教程。
要记住 word2vec 的维度本身没有任何特定含义（不像大五人格中的“外向性”等），一切都“取决于旋转”。直接使用它看起来很笨拙。另外在实现方面，我很惊讶文章没提到这个 PyTorch 版 word2vec 教程，在众多实现（包括 NumPy 和 TF 的）中，它是最直接、最具教学性的。
stared查看英文原文
Well, I think that is important to remember that dimensions of word2vec DO NOT have any specific meaning (unlike Extraversion etc in Big Five). All of it is "up to a rotation". Using it looks clunky at best. To be fair, I may be biased as I wrote a different intro to word2vec (http://p.migdal.pl/2017/01/06/king-man-woman-queen-why.html). For implementation, I am surprised it leaves out https://adoni.github.io/2017/11/08/word2vec-pytorch/. There are many other, including in NumPy and TF, but I find the PyTorch one the most straightforward and didactic, by a large margin.
提出疑问：一词多义（如 polish）在向量模型中如何表示。
好文章，谢谢。我思考 word2vec 时的困惑是：向量模型如何存储那些有多个差异极大含义的词的信息，比如“polish”既指波兰的，也指擦亮的。
rmbryan查看英文原文
Excellent article, thank you. My snag in thinking about word2vec is how the vector model stores information about words with multiple, significantly different definitions, such as 'polish', Eastern Europe or glistening clean.
历次讨论
同类还值得读
如何写一个拼写纠正器
How to Write a Spelling Corrector
Peter Norvig 用一个几十行的 Python 程序演示如何做拼写纠正：先用编辑距离（删除、插入、替换、换位）生成所有可能的候选词，再用从大量文本中统计出的词频概率，选出…
AI 与机器学习上首页 10 次 · 2007–2022norvig.com编辑精选面向黑客的概率编程与贝叶斯方法
Probabilistic Programming and Bayesian Methods for Hackers (2013)
《黑客的贝叶斯方法与概率编程》是一本开源的贝叶斯统计入门书，以 Python/PyMC 代码和交互式 Jupyter Notebook 为主线，强调“先实践、后数学”的学习路径，…
AI 与机器学习上首页 5 次 · 2013–2018camdavidsonpilon.github.io编辑精选神经网络黑客指南
Hacker's Guide to Neural Networks
Andrej Karpathy 写的神经网络入门教程，面向程序员，从最简单的表达式求导讲起，逐步推导出反向传播和梯度下降，最后引出神经网络。
AI 与机器学习上首页 4 次 · 2014–2019karpathy.github.io编辑精选神经网络能逼近任意函数的可视化证明
A visual proof that neural nets can approximate any function
这一章来自 Michael Nielsen 的在线书《Neural Networks and Deep Learning》，用 sigmoid 单元叠加成局部隆起/平滑阶跃的图示…
AI 与机器学习上首页 4 次 · 2014–2022neuralnetworksanddeeplearning.com编辑精选论乔姆斯基与统计学习的两种文化
On Chomsky and the Two Cultures of Statistical Learning
Norvig 回应 Chomsky 对统计语言模型批评的文章，围绕“统计学习能否解释语言”展开，比较生成与解释、少参数模型与 Shannon 式高参数统计模型。
AI 与机器学习上首页 4 次 · 2011–2025norvig.com编辑精选