AI 与机器学习ai.stanford.edu
机器学习为什么“难”？
Why is machine learning ‘hard’?
2016 — 2024，它一次次回到首页
黑点是上首页（≥ 50 分）的那次投稿，空心点是没冲上去的投稿，点越大分数越高。
导读
斯坦福研究者 Zayd Enam 的这篇文章探讨机器学习为什么“难”：与普通编程不同，ML 的失败往往不是报错，而是模型效果不佳，且原因难以定位。作者指出调试 ML 系统需要理解数据、模型与算法之间的复杂交互，这使其本质上比传统软件开发更难把握。
为什么经久不衰：它把 ML 工程中“效果不好却不知为何”的普遍痛点讲得透彻，从 2016 到 2024 两次登上 HN 首页，说明这一困惑至今仍未过时。
HN 评论精选
ML 缺乏可依据的理论来指导模型选择，实践全靠试错和直觉
我认为问题在于我们还没真正理解机器学习。比如选超参数——我无数次问专家、培训师和同事：“我该怎么知道该用什么模型？几层？每层多少节点？要不要 dropout？”答案永远是“多试几种，挑效果最好的”。这让我觉得奇怪又不安：我们似乎还没能针对给定数据集明确说出需要什么样的模型。考虑到教 ML 时抛给你那么多数学、证明和算法，实际动手时却全靠“直觉”（猜测），实在让人不爽。
mattlondon查看英文原文
I think the problem is that we don't really understand ML properly yet. Like picking hyperparamters - time and time again I've asked experts/trainers/colleagues: "How do I know what type of model to use? How many layers? How many nodes per layer? Dropout or not?" etc etc And the answer is always along the lines of "just try a load of stuff and pick the one that works best". To me, that feels weird and worrying. Its like we don't yet understand ML properly yet to definitively say, for a given data set, what sort of model we'll need. This can lead us down the debugging black-hole TFA talks about since we appear to have zero-clue about why we chose something, so debugging something ultimately might just be "opps - we chose 3 layers of 10, 15, and 11 nodes, instead of 3 layers of 10, 15 and 12 nodes! D'oh! Lets start training again!" It really grates me to think about this considering how much maths and proofs and algorithms get thrown at you when being taught ML, then to be told when it comes to actually doing something its all down to "intuition" (guessing). And yeah as others have said - data :-)
数值编程普遍难调试，ML 的困难是连续数学难以推理的体现
某种程度上，这种困难在任何数值编程中都存在。典型软件工程的 bug 通常在逻辑里，绝大多数问题用可视化调试器就能诊断。而数值代码中，你往往无法靠查看几个离散变量的状态和堆栈来找出问题。“为什么这个矩阵奇异？”可能要花好几天。你花大量时间盯着代码、对照纸上的数学、试图可视化高维中间数据。连续数学比离散的难推理得多。
blt查看英文原文
I think to some extent, this kind of difficulty occurs with any numerical programming. With typical software engineering the bugs are usually in logic. All but the hardest problems can be diagnosed with a visual debugger. With numerical code, though, you usually can't look at the state of a few discrete-valued variables and a stack trace to figure out what went wrong. "Why is this matrix singular?" can take days to answer. You spend a lot of time staring at the code, comparing it to the math on paper, trying to visualize high-dimensional intermediate data, etc. Continuous math can be a lot harder to reason about than discrete.
给出实用方法论：从朴素基线起步、逐步加复杂度、定期化简、重视数据
用 ML 解决问题有一个贪心策略：1. 从一个经过验证的朴素模型开始，建立可回退的基线，比如先全连接网络，再朴素 CNN，逐步加 BN、ReLU、残差连接等。2. 不要花太多时间调超参数，尤其在深度学习中，一旦改了网络结构，一切都变了。3. 逐步增加复杂度：有了扎实基线后，再往里加花哨的想法，你会发现它们只是对已有效方案的改进，加起来并不难。4. 随着时间推移，改动之间可能互相冲突，所以要定期化简，重新审视方法，去掉不再合适的东西。5. 看你的数据。垃圾进，垃圾出。看看你自己作为最智能的存在能否理解这些数据，如果不能，就该提升数据质量了。
eva1984查看英文原文
So there is a greedy strategy to approach problem with ML 1.Starts with a VANILLA model, a proven one. To establish a baseline you can fall back on. For example, in deep learning, starts with fully-connected nets, then vanilla CNN, adding BNs and ReLus, then residual connections, etc. 2.Do not spend too much time tuning hyperparmeters, especially in the field of deep learning, once you change the your algorithm, a.k.a network structure, everything changes. 3.Adding complexity as you go. It is important once you established some solid baseline, then you can start add more fancy ideas into your stack, and you will find, fancy ideas are improvement over the already working ideas, and it is not that hard to add it. 4.One important tip to remind, once you change your algorithm, as time goes, those changes might not happy with each other. So reduction is also very important. Rethink your approach from time to time, take away stuff that didn't fit anymore. 5.Look At Your Data. Garbage in, Garbage out. Cannot be more true. Really. Look at your data, maybe a sample of it, see whether youself, as the most intelligent being as of yet, can make sense of it or not. If you cannot, then you need to probably improve the quality. Anyway, ML is a very complex field and developing like crazy, but I didn't feel the methodology to tackle it is any different from any other complex problems. It is a iterative processes starting from simple prove solutions to something greater, piece by piece. Watch and think then improve.
ML 系统还带来大量架构与技术债问题，推荐相关论文
机器学习组件还有很多架构层面的问题需要考虑，Sculley 等人的《机器学习：高利率的技术债务信用卡》一文对此有精彩总结。
jupp0r查看英文原文
There are also lots of architectural problems with machine learning components to consider, beautifully summarized in "Machine Learning: The High Interest Credit Card of Technical Debt" by Sculley et al. http://research.google.com/pubs/pub43146.html
历次讨论
同类还值得读
如何写一个拼写纠正器
How to Write a Spelling Corrector
Peter Norvig 用一个几十行的 Python 程序演示如何做拼写纠正：先用编辑距离（删除、插入、替换、换位）生成所有可能的候选词，再用从大量文本中统计出的词频概率，选出…
AI 与机器学习上首页 10 次 · 2007–2022norvig.com编辑精选面向黑客的概率编程与贝叶斯方法
Probabilistic Programming and Bayesian Methods for Hackers (2013)
《黑客的贝叶斯方法与概率编程》是一本开源的贝叶斯统计入门书，以 Python/PyMC 代码和交互式 Jupyter Notebook 为主线，强调“先实践、后数学”的学习路径，…
AI 与机器学习上首页 5 次 · 2013–2018camdavidsonpilon.github.io编辑精选神经网络黑客指南
Hacker's Guide to Neural Networks
Andrej Karpathy 写的神经网络入门教程，面向程序员，从最简单的表达式求导讲起，逐步推导出反向传播和梯度下降，最后引出神经网络。
AI 与机器学习上首页 4 次 · 2014–2019karpathy.github.io编辑精选神经网络能逼近任意函数的可视化证明
A visual proof that neural nets can approximate any function
这一章来自 Michael Nielsen 的在线书《Neural Networks and Deep Learning》，用 sigmoid 单元叠加成局部隆起/平滑阶跃的图示…
AI 与机器学习上首页 4 次 · 2014–2022neuralnetworksanddeeplearning.com编辑精选论乔姆斯基与统计学习的两种文化
On Chomsky and the Two Cultures of Statistical Learning
Norvig 回应 Chomsky 对统计语言模型批评的文章，围绕“统计学习能否解释语言”展开，比较生成与解释、少参数模型与 Shannon 式高参数统计模型。
AI 与机器学习上首页 4 次 · 2011–2025norvig.com编辑精选