AI 与机器学习arxiv.org
经梯度下降学到的每个模型都近似于核机器
Every Model Learned by Gradient Descent Is Approximately a Kernel Machine
2020 — 2024，它一次次回到首页
黑点是上首页（≥ 50 分）的那次投稿，空心点是没冲上去的投稿，点越大分数越高。
导读
Pedro Domingos 的这篇论文证明：任何用梯度下降（SGD）训练得到的模型，无论架构如何，都近似等价于一个使用特定“路径核”的核机器（如 SVM）。路径核衡量的是，在训练过程中，模型在两个数据点上的梯度变化有多相似。这是一种理论上的形式等价，揭示了 SGD 训练与经典核方法之间出人意料的深层联系。
为什么经久不衰：它把看似迥异的深度学习与已被充分理解的核方法联系起来，为理解神经网络的学习行为提供了新视角，同时评论也冷静指出其计算意义有限。
HN 评论精选
质疑这类发现的意义：通用逼近器很多，难点从来是训练而非架构，理论上能互相近似并不意外。
我对这类发现感到困惑，希望有人能给我科普一下。已知的通用逼近器很多：深度网络是一种，SVM 是一种，三次样条也是一种，而且它们已经用了快一百年。问题从来不是找到足够强的逼近器，而是训练它。据我理解，深度学习的重大进展在于我们终于搞清楚了如何训练某一类通用逼近器，使其能为过去无法解决的分类问题找到非常好的分离面。但理论上存在其他通用逼近器能近似重现相同的分离面，这不该让人惊讶吧？任何通用逼近器都应该足够强大到能重现这些分离面——“通用”这个词就是这个意思。问题一直是找到正确的权重，而不是找到正确的逼近器架构。我是不是漏了什么？
GlenTheMachine查看英文原文
I'm confused by findings like this one, and I'm hoping someone here can educated me. There are many known universal approximations. Deep networks are one. SVMs are one. Heck, cubic splines are one, and they've been in use for nearly a hundred years IIRC. The problem has never been one of finding a sufficiently powerful approximator. It has been training that approximator. My understanding of the significant advancement made by deep learning is that we finally figured out how to train a specific kind of universal approximator in a way such that it finds very good separation surfaces for what used to be impossible-to-solve classification problems. But it should be no surprise to anyone that there exist, in theory, other universal approximations that approximately reproduce the same separation surfaces, should it? I'd expect any universal approximator to be powerful enough to reproduce the separation surfaces, hence the meaning of the word "universal". The problem was always finding the right weights, not finding the right approximator architecture. Am I missing something?
解释路径核的含义，并指出该工作的价值在于揭示 SGD 模型与 SVM 之间的深层理论联系。
作者指出，任何用 SGD 训练的模型，不论架构，都近似等价于一个具有特定类型核的核机器（即 SVM），Domingos 称之为“路径核”。按论文定义，路径核衡量任意两个数据点上模型变化（具体说是模型梯度变化）的相似程度。这不是常见的径向基核。我们早就知道 SVM 是通用逼近器，理论上能逼近任何目标函数。这项工作的意义在于，它发现了任何经 SGD 训练的模型与已被充分理解的 SVM 之间一个新的、令人惊讶的深层联系。
cs702查看英文原文
"Here we show that every model learned by this method [SGD], regardless of architecture, is approximately equivalent to a kernel machine [i.e., a support vector machine or SVM] with a particular type of kernel" -- a type of kernel which Domingos, the author, calls a "path kernel." As defined in the paper, a "path kernel" measures, for any two data points, how similarly a model varies (specifically, how similarly the model's gradients change) at those two data points during training via SGD. This isn't exactly your usual, plain-vanilla, radial-basis type of kernel. We've known for a long time that SVMs are universal approximators, i.e., in theory they can approximate any target function. The importance of this work is that it has found a new, surprising, deep connection between any model trained via SGD and SVMs, which are well understood :-)
指出这只是形式等价而非计算实用——路径核和系数都涉及路径积分，推理代价约等于重新训练。
对那些说“哇，我们可以回到 SVM 了，巨大加速”的人——这有点为时过早。这只是形式上的等价，计算上并不实用。这里的数学基本是本科一年级微积分水平，第 2 节写得很清楚，值得一读。作者本质上证明的是，任何用“无限小步长全批量梯度下降”（即沿梯度流）训练的模型都能写成核化形式 y(x)=Σa_i K(x,x_i)+b。人们对 SVM 的直觉是 a_i 是常数、稀疏的，核函数 K 计算便宜——这也是“核技巧”的由来。但这些性质在这里都不成立，所以它在计算上并不令人兴奋。诀窍在于 a_i 和 K 都涉及沿梯度流的路径积分，所以单次推理的代价大约等于从头训练模型。
ajtulloch查看英文原文
For everyone saying "oh wow, we can go back to SVMs now, huge speedups, etc" - that's more than a little premature. This is purely a formal equivalence, but not very useful computationally. The math here is pretty much first-year undergraduate level calculus, and it's worth going through section 2 since it's quite clearly written (thanks Prof Domingos). Essentially, what the author does is show that any model trained with "infinitely-small step size full-batch gradient descent" (i.e. a model following a gradient flow) can be written in a "kernelized" form y(x) = \sum_{(x_i, y_i) \in L} a_i K(x, x_i) + b. The intuition most people have for SVMs is that the constants a_i are, well, constant, that the a_i are sparse, and that the kernel function K(x, x_i) is cheap to compute (partly why it's called the 'kernel trick'). However, none of those properties apply here, which is why this isn't particularly exciting computationally. The "trick" is that both a_i and K(x, x_i) involve path integrals along the gradient flow for the input x, and so doing a single inference is approximately as expensive as training the model from scratch (if I understand this correctly).
引用相关研究，认为 DNN 的优势主要来自更好的训练/微调，而非根本不同的决策策略。
另见 Brendel 和 Bethge 的论文“Approximating CNNs with bag-of-local-features models works surprisingly well on ImageNet”（本文引用了它），该文表明 DNN 相比以往 bag-of-feature 分类器的进步，主要来自更好的微调，而非质上不同的决策策略。
sarosh查看英文原文
See also W. Brendel and M. Bethge. "Approximating CNNs with bag-of-local-features models works surprisingly well on ImageNet." https://arxiv.org/abs/1904.00760 (which is referenced in this paper) and explains "[t]his suggests that the improvements of DNNs over previous bag-of-feature classifiers in the last few years is mostly achieved by better fine-tuning rather than by qualitatively different decision strategies."
历次讨论
同类还值得读
如何写一个拼写纠正器
How to Write a Spelling Corrector
Peter Norvig 用一个几十行的 Python 程序演示如何做拼写纠正：先用编辑距离（删除、插入、替换、换位）生成所有可能的候选词，再用从大量文本中统计出的词频概率，选出…
AI 与机器学习上首页 10 次 · 2007–2022norvig.com编辑精选面向黑客的概率编程与贝叶斯方法
Probabilistic Programming and Bayesian Methods for Hackers (2013)
《黑客的贝叶斯方法与概率编程》是一本开源的贝叶斯统计入门书，以 Python/PyMC 代码和交互式 Jupyter Notebook 为主线，强调“先实践、后数学”的学习路径，…
AI 与机器学习上首页 5 次 · 2013–2018camdavidsonpilon.github.io编辑精选神经网络黑客指南
Hacker's Guide to Neural Networks
Andrej Karpathy 写的神经网络入门教程，面向程序员，从最简单的表达式求导讲起，逐步推导出反向传播和梯度下降，最后引出神经网络。
AI 与机器学习上首页 4 次 · 2014–2019karpathy.github.io编辑精选神经网络能逼近任意函数的可视化证明
A visual proof that neural nets can approximate any function
这一章来自 Michael Nielsen 的在线书《Neural Networks and Deep Learning》，用 sigmoid 单元叠加成局部隆起/平滑阶跃的图示…
AI 与机器学习上首页 4 次 · 2014–2022neuralnetworksanddeeplearning.com编辑精选论乔姆斯基与统计学习的两种文化
On Chomsky and the Two Cultures of Statistical Learning
Norvig 回应 Chomsky 对统计语言模型批评的文章，围绕“统计学习能否解释语言”展开，比较生成与解释、少参数模型与 Shannon 式高参数统计模型。
AI 与机器学习上首页 4 次 · 2011–2025norvig.com编辑精选