AI 与机器学习distill.pub编辑精选
动量法为何有效（Distill 互动讲解）
Why Momentum Works
2017 — 2025，它一次次回到首页
黑点是上首页（≥ 50 分）的那次投稿，空心点是没冲上去的投稿，点越大分数越高。
导读
这是 Distill 的交互式讲解文章，用可视化和推导解释梯度下降中的动量法为什么有效：它把历史梯度累积成“速度”，在方向反复变化的维度上相互抵消，在持续一致的维度上加速前进。文章帮助读者从直觉上理解动量如何影响优化过程和收敛。
为什么经久不衰：交互式图解和清晰推导把动量的直觉与数学一次讲透，是理解深度学习优化器历久弥新的经典入门。
HN 评论精选
质疑文章动量形式的归一化，指出它除记忆效应外还带来 1/(1-B) 的乘数效应，想区分两者对优化的贡献。
我很好奇给梯度赋予短期记忆所选择的方法。我见过的时间序列常见做法是 Y[i+1]=B*Y[i]+(1-B)*X[i+1]，其中 0<=B<=1；如果 X 变为常数，Y 会收敛到该常数。而文章给梯度加短期记忆的方式是 Y[i+1]=B*Y[i]+X[i+1]；如果 X 变为常数，Y 会收敛到 X/(1-B)。短期记忆似乎不能准确描述它在做什么：里面确实有记忆效应，但在输入不变的区域还有乘数效应。所以我好奇改进有多少来自记忆效应，多少来自乘数效应？更常见的 B 和 1-B 加权是否也有助于梯度下降？
tzs查看英文原文
I'm curious about the method chosen to give short term memory to the gradient. The most common way I've seen when people have a time sequence of values X[i] and they want to make a short term memory version Y[i] is to do something of this form: Y[i+1] = B * Y[i] + (1-B) * X[i+1] where 0 <= B <= 1. Note that if the sequence X becomes a constant after some point, the sequence Y will converge to that constant (as long as B != 1). For giving the gradient short term memory, the article's approach is of the form: Y[i+1] = B * Y[i] + X[i+1] Note that if X becomes constant, Y converges to X/(1-B), as long as B in [0,1). Short term memory doesn't really seem to describe what this is doing. There is a memory effect in there, but there is also a multiplier effect when in regions where the input is not changing. So I'm curious how much of the improvement is from the memory effect, and how much from the multiplier effect? Does the more usual approach (the B and 1-B weighting as opposed to a B and 1 weighting) also help with gradient descent?
赞赏 Distill 量化作者贡献的做法，并希望推广到学术论文。
Distill 上一些多作者文章有一个（我认为）非常重要的创新：它们精确量化了每位作者对文章的贡献。我希望这成为科学论文的常态——一方面，如果出现问题，能清楚该问谁；另一方面，各种大人物也不会在几乎没参与科学工作的论文作者名单里获得荣誉位置。
throwaway71958查看英文原文
Some of the multi-author articles on Distill have a very important (IMO) innovation. They quantify precisely the contribution each author has made to the article. I would like to see this become norm in scientific papers, so on the one hand it'd be clear who to ask questions if they arise, and on the other the various dignitaries won't get their honorary spot on the authors list of papers they were barely involved with scientifically.
历次讨论
同类还值得读
如何写一个拼写纠正器
How to Write a Spelling Corrector
Peter Norvig 用一个几十行的 Python 程序演示如何做拼写纠正：先用编辑距离（删除、插入、替换、换位）生成所有可能的候选词，再用从大量文本中统计出的词频概率，选出…
AI 与机器学习上首页 10 次 · 2007–2022norvig.com编辑精选面向黑客的概率编程与贝叶斯方法
Probabilistic Programming and Bayesian Methods for Hackers (2013)
《黑客的贝叶斯方法与概率编程》是一本开源的贝叶斯统计入门书，以 Python/PyMC 代码和交互式 Jupyter Notebook 为主线，强调“先实践、后数学”的学习路径，…
AI 与机器学习上首页 5 次 · 2013–2018camdavidsonpilon.github.io编辑精选神经网络黑客指南
Hacker's Guide to Neural Networks
Andrej Karpathy 写的神经网络入门教程，面向程序员，从最简单的表达式求导讲起，逐步推导出反向传播和梯度下降，最后引出神经网络。
AI 与机器学习上首页 4 次 · 2014–2019karpathy.github.io编辑精选神经网络能逼近任意函数的可视化证明
A visual proof that neural nets can approximate any function
这一章来自 Michael Nielsen 的在线书《Neural Networks and Deep Learning》，用 sigmoid 单元叠加成局部隆起/平滑阶跃的图示…
AI 与机器学习上首页 4 次 · 2014–2022neuralnetworksanddeeplearning.com编辑精选论乔姆斯基与统计学习的两种文化
On Chomsky and the Two Cultures of Statistical Learning
Norvig 回应 Chomsky 对统计语言模型批评的文章，围绕“统计学习能否解释语言”展开，比较生成与解释、少参数模型与 Shannon 式高参数统计模型。
AI 与机器学习上首页 4 次 · 2011–2025norvig.com编辑精选