AI 与机器学习colah.github.io编辑精选
神经网络、流形与拓扑
Neural Networks, Manifolds, and Topology
2014 — 2019，它一次次回到首页
黑点是上首页（≥ 50 分）的那次投稿，空心点是没冲上去的投稿，点越大分数越高。
导读
这是 Chris Olah（colah.github.io）的经典长文，用大量可视化从几何与拓扑角度解释神经网络：把神经网络看作对输入空间的一系列连续变形，分类任务本质上是把纠缠在一起的低维流形解开，直到能用超平面分开。文章还讨论了流形假设，以及某些拓扑结构（如环面上的圆圈）为何会让浅层网络难以学习。
为什么经久不衰：它用直观的几何直觉把神经网络'为什么有效、何时会失效'讲得异常清晰，可视化精美且门槛低，多年来反复被推荐为理解深度学习的最佳入门读物之一。
HN 评论精选
好奇神经网络学习难度的理论边界，并引用对抗样本论文说明深度网络学习的病态性
这是一个非常有趣的神经网络视角，是新的吗？我从没见过对神经网络的几何解释。尤其是这句：'显然判断纽结是否平凡是 NP 问题，这对神经网络来说不是好兆头。'有没有关于神经网络学习障碍的理论研究？虽然不会改变实践（比如高斯混合的 MLE 学习是 NP 难的，但大家照样在用），但或许能揭示'困难'实例的几何结构。另外，最近一篇论文表明，给图像分类网络加上精心构造但肉眼不可见的扰动，预测结果会完全改变。
ot查看英文原文
This is a very interesting perspective on neural networks, is it novel? I've never seen a geometric interpretation of NNs. In particular this: > (Apparently determining if knots are trivial is NP. This doesn’t bode well for neural networks.) Is there any theoretical research on obstructions to NN learning? Not that it would change much the practice (for instance, MLE learning of gaussian mixtures is NP-hard, but everybody does it anyway), but it could shed some light on the geometry of the "hard" instances. EDIT: for example, to get a sense of how ill-conditioned is deep NN learning, a recent paper [1] shows that if we feed to an object classifying NN an image with crafted but unnoticeable perturbations the predictions change completely. [1] http://arxiv.org/abs/1312.6199
认为流形复合体的视角切中要害，但相应的学习器尚未被真正构建出来
太精彩了。我一直期待一种能跳出浮点向量表示的机器学习方法，因为我觉得它并不能很好地描述自然发生的问题，大多数本体是半结构化的。而这篇文章触及了问题的核心：真实但可解的问题是流形复合体。不过我认为还没有人真正为流形复合体构建过学习器，持续同调算是朝正确方向的一次尝试，但在我看来还不够。
tlarkworthy查看英文原文
Bloody brilliant. I have longed for a ML methodology that breaks out of the float vector representation because I don't think it describes naturally occurring problems that well, most ontologies are semi-structured. However, this really gets to the nub of the problem, real, but solvable, problems are manifold complexes. I don't think anyone has really built a learner for complexes though, persistent homology is a stab in the right direction but its not there yet IMHO.
认为原文的证明太含糊，并给出了一个更严格的线性代数证明
我不喜欢他关于 1-2 个隐藏单元的系统无法处理蓝色环面中的红色圆圈的证明，感觉太含糊。我的证明：假设矩阵 W 是奇异的，那么存在非零向量 v 使得 Wv=0。取红圈上一点 x，再找另一个点 y=x+av（a 为某个标量）落在蓝色环面上。则 Wy+b=W(x+av)+b=Wx+aWv+b=Wx+b。证毕。
yummyfajitas查看英文原文
I don't like his proof that a 1-2 hidden unit system can't handle the red circle in the blue torus. Feels too handwavy. My proof: Suppose the matrix W is singular. Then there exists a nonzero vector v for which W v = 0. Find a point x in the red circle, then find a different point y=x+av (for some scalar a) in the blue torus. Wy+b=W(x+av)+b=Wx+aWv+b=Wx+a0+b=wX+b. Fin.
用流形假设解释为什么实践中较小的神经网络也能有很强的预测能力
之前的讨论：……但那边讨论不多。这些可视化非常棒，基本刷新了我的认知。我此前不知道流形假设：自然数据在其嵌入空间中形成低维流形，有理论和实验上的理由相信这一点。如果你接受这个假设，那么分类算法的任务从根本上就是分离一堆纠缠的流形。我的解读是：区分猫狗图片在最坏情况下似乎需要巨大的网络，但现实中猫狗图像可以被'重排'成一种便于解开的形状，且这些形状可能存在于低得多的维度中。这或许能（在某种意义上）解释为什么相对较小的神经网络有如此强的预测能力。
flancian查看英文原文
Previously: https://news.ycombinator.com/item?id=7557964 https://news.ycombinator.com/item?id=9814114 But not a lot of discussion over there. The visualizations are great, and this basically blew my mind. I didn’t know of the manifold hypothesis until now. The manifold hypothesis is that natural data forms lower-dimensional manifolds in its embedding space. There are both theoretical and experimental reasons to believe this to be true. If you believe this, then the task of a classification algorithm is fundamentally to separate a bunch of tangled manifolds. My interpretation/rephrasing: if you want to build a neural network that distinguishes cat and dog pictures, in the worst case that would seem to require a huge network with many nodes/layers (say, the number being a function of the size of the image) rather than the number that seems to work reasonably well in practice (six or some other rather low constant number observed in reality). So the number of dimensions over which the “images” are potentially spread is huge, but it’d seem that in the real world one can rearrange the dog and cat images in a “shape” that then allows for relatively easy disentanglement by the neural network; and these shapes can probably be realized in much lower dimensions (in the example, six). This could explain (for some definition of explain) the observed predictive power of relatively small neural networks.
历次讨论
同类还值得读
如何写一个拼写纠正器
How to Write a Spelling Corrector
Peter Norvig 用一个几十行的 Python 程序演示如何做拼写纠正：先用编辑距离（删除、插入、替换、换位）生成所有可能的候选词，再用从大量文本中统计出的词频概率，选出…
AI 与机器学习上首页 10 次 · 2007–2022norvig.com编辑精选面向黑客的概率编程与贝叶斯方法
Probabilistic Programming and Bayesian Methods for Hackers (2013)
《黑客的贝叶斯方法与概率编程》是一本开源的贝叶斯统计入门书，以 Python/PyMC 代码和交互式 Jupyter Notebook 为主线，强调“先实践、后数学”的学习路径，…
AI 与机器学习上首页 5 次 · 2013–2018camdavidsonpilon.github.io编辑精选神经网络黑客指南
Hacker's Guide to Neural Networks
Andrej Karpathy 写的神经网络入门教程，面向程序员，从最简单的表达式求导讲起，逐步推导出反向传播和梯度下降，最后引出神经网络。
AI 与机器学习上首页 4 次 · 2014–2019karpathy.github.io编辑精选神经网络能逼近任意函数的可视化证明
A visual proof that neural nets can approximate any function
这一章来自 Michael Nielsen 的在线书《Neural Networks and Deep Learning》，用 sigmoid 单元叠加成局部隆起/平滑阶跃的图示…
AI 与机器学习上首页 4 次 · 2014–2022neuralnetworksanddeeplearning.com编辑精选论乔姆斯基与统计学习的两种文化
On Chomsky and the Two Cultures of Statistical Learning
Norvig 回应 Chomsky 对统计语言模型批评的文章，围绕“统计学习能否解释语言”展开，比较生成与解释、少参数模型与 Shannon 式高参数统计模型。
AI 与机器学习上首页 4 次 · 2011–2025norvig.com编辑精选