AI 与机器学习neuralnetworksanddeeplearning.com编辑精选
神经网络能逼近任意函数的可视化证明
A visual proof that neural nets can approximate any function
2014 — 2022，它一次次回到首页
黑点是上首页（≥ 50 分）的那次投稿，空心点是没冲上去的投稿，点越大分数越高。
导读
这一章来自 Michael Nielsen 的在线书《Neural Networks and Deep Learning》，用 sigmoid 单元叠加成局部隆起/平滑阶跃的图示，直观说明单隐层神经网络可以逼近任意连续函数。它讲的是存在性与构造直觉，而不是训练一定能找到有用函数，也不是计算意义上的通用性。
为什么经久不衰：它把“万能逼近定理”讲成可拼接的图形直觉，并持续引发关于近似质量与适用边界的讨论，所以今天仍值得读。
HN 评论精选
强调定理只适用于连续函数的逼近，不等于通用计算能力。
是逼近，不是计算。函数还必须是连续的。神经网络适合逼近、插值和某些情况下的外推，在某些问题领域很有用；但这当然不使它成为某种通用计算机（在可计算性意义上，像通用图灵机那样）。
a-nikolaev查看英文原文
Approximate, not compute. The function also must be continuous. NNs are good for approximation / interpolation / extrapolation, which makes them quite useful for certain domains of problems. But of course, it does not make them a kind of universal computing machine (in the computability sense, like universal Turing machines).
万能逼近不是神经网络独有；按误差界、效率和导数逼近等标准，它常不如传统近似方法。
拉格朗日多项式、傅里叶级数等也都能做到。真正的问题是逼近得好不好：能否证明误差界？能否界定最大误差？是否高效（低存储、低计算量）？构建是否快？对导数和梯度的逼近如何、误差是否有界？从近似论的几乎每个方面看，神经网络都是逼近连续函数最差的方法之一；类比排序算法，它们比 bogosort 还差。没有误差界，不能界定最大误差，求系数很慢（训练需要 GPU 等），求值需要大量存储和算力……
fluffything查看英文原文
So can any lagrange polynomial, Fourier series, ... The real question is whether the approximation is a good one: - can you prove error bounds ? - can you bound the maximum error? - is it efficient ? (low storage, low computational effort) - is it fast to build? (low computational effort of coefficients) - derivatives: how well does it approximate gradients, what's the error on the gradient, is it bounded? can one bound it, how fast can one evaluate them, etc. - there are many other interesting properties: https://en.wikipedia.org/wiki/Approximation_theory From pretty much every single aspect of approximation theory, neural nets are one of the worst methods to approximate a continuous function. If you were to make an analogy with sorting algorithms, they would be worse than bogosort. There are no error bounds, you can't bound the maximum error, computing their coefficients is very slow (training, needs GPUs, ...), they require a lot of storage and computational power to evaluate, ...
万能逼近是通用建模的必要条件而非充分条件，不应被当成超能力。
很多评论对神经网络的万能逼近义愤填膺，我认为问题在于它常被说成超能力，而不是任何通用算法的基础门槛。我主张，任何不具备万能逼近性质的建模技术，无论投入多少功夫（比如特征工程），都注定在一大类问题上失败。也就是说，UA 是一种建模技术完全通用（即能成为 AGI 基础）的必要但不充分条件。
kevinventullo查看英文原文
I see a lot of commenters up in arms about Universal Approximation for NN’s, and I think the issue is that it’s often framed as a superpower rather than table stakes for any kind of general purpose algorithm. I posit that any modeling technique which does not have the universal approximation property will be guaranteed to fail on large classes of problems no matter how much elbow grease (say feature engineering) one puts into it. That is, UA is a necessary but not sufficient condition for a modeling technique to be fully general (i.e. could form the basis of whatever AGI is).
批评多偏离文章教学目的，它作为基础直觉入门仍然有价值。
我不理解这里这么多批评。这是神经网络基础入门第 4 章。作者用通俗意义上的“逼近”一词，提供了写得好、直观且简洁的演示，说明 sigmoid 如何拼起来逼近函数。大约 5 年前我读到它时，它帮我建立了直觉。还有其他好章节讲梯度消失或反向传播。这里的批评大多甚至不是文章主题：“它不能计算不可计算函数”“它不能预测明天股价”“样条更高效”“它不能预测水中的棍子看起来怎样”。这就像说“我读了两遍《禅与摩托车维修艺术》，还是不知道怎么调火花塞。烂书”。
nautilius查看英文原文
I don't understand all of the criticism here. This is Ch. 4 in a basic intro to neural networks. The author provides a well written, intuitive, and concise demonstration of how sigmoids can be pieced together to approximate functions, in the layman's sense of the word. It helped me build intuition when I came across this maybe 5 (?) years ago. There are other good 'Chapters' about vanishing gradient or backprop. The criticism here is mostly about things that aren't even the topic of the article: 'it cannot compute non-computable functions', 'it cannot predict tomorrow's stock price', 'splines are more efficient', 'it cannot predict how a stick looks bent in water'. It's like saying 'I read "Zen and the art of motorcycle maintenance, twice, and still don't know how to adjust spark plugs. Stupid book"'.
历次讨论
同类还值得读
如何写一个拼写纠正器
How to Write a Spelling Corrector
Peter Norvig 用一个几十行的 Python 程序演示如何做拼写纠正：先用编辑距离（删除、插入、替换、换位）生成所有可能的候选词，再用从大量文本中统计出的词频概率，选出…
AI 与机器学习上首页 10 次 · 2007–2022norvig.com编辑精选面向黑客的概率编程与贝叶斯方法
Probabilistic Programming and Bayesian Methods for Hackers (2013)
《黑客的贝叶斯方法与概率编程》是一本开源的贝叶斯统计入门书，以 Python/PyMC 代码和交互式 Jupyter Notebook 为主线，强调“先实践、后数学”的学习路径，…
AI 与机器学习上首页 5 次 · 2013–2018camdavidsonpilon.github.io编辑精选神经网络黑客指南
Hacker's Guide to Neural Networks
Andrej Karpathy 写的神经网络入门教程，面向程序员，从最简单的表达式求导讲起，逐步推导出反向传播和梯度下降，最后引出神经网络。
AI 与机器学习上首页 4 次 · 2014–2019karpathy.github.io编辑精选论乔姆斯基与统计学习的两种文化
On Chomsky and the Two Cultures of Statistical Learning
Norvig 回应 Chomsky 对统计语言模型批评的文章，围绕“统计学习能否解释语言”展开，比较生成与解释、少参数模型与 Shannon 式高参数统计模型。
AI 与机器学习上首页 4 次 · 2011–2025norvig.com编辑精选语音与语言处理（第三版草稿，2017）
Speech and Language Processing, 3rd ed. draft (2017)
Jurafsky 与 Martin 经典教材《Speech and Language Processing》第三版草稿的官方页面，按章提供 PDF，覆盖语音识别、自然语言处理与语…
AI 与机器学习上首页 4 次 · 2018–2025web.stanford.edu编辑精选