AI 与机器学习idlewords.com编辑精选
超级智能：一个吞噬聪明人的想法
Superintelligence: The Idea That Eats Smart People
2016 — 2018，它一次次回到首页
黑点是上首页（≥ 50 分）的那次投稿，空心点是没冲上去的投稿，点越大分数越高。
导读
Maciej Cegłowski 的演讲文本，用讽刺和“外部视角”审视“超级智能”风险论：它逐一拆解常见类比，质疑把智能当成可无限放大的 CPU 速度，并担心这套话语会让聪明人放弃常识。文章重点不在预测 AI 时间表，而在批评论证方式、群体动力学和科幻式想象。
为什么经久不衰：十多年后 AI 话题更热，这篇把风险论证当作修辞和社群现象来检查的文章，仍是防止被宏大叙事带走的清醒剂。
HN 评论精选
AI 风险更可能来自军事滥用、经济不平等和现有技术，而非科幻式超级智能。
我同意 Maciej 的核心观点，但觉得他给出的内部论证相当弱。即使接受 AI 风险派的假设，我也不认为 AI 风险紧迫。真正的风险不是全知全能、把“让地球所有人快乐”误解成毁灭地球的学者型 AI，而是正确执行“杀死某群体”命令的军用 AI——高效到主人开始想下一群体、再下一个；也是智能工厂在新兴极小超级阶层与贫困大众之间撕开巨大鸿沟。AI 远非唯一能把危险人物变成生存威胁的技术：核武器已存在且越来越便宜，收入不平等已在飙升，互联网也催生了史上最强监控与宣传工具。
alexbecker查看英文原文
While I agree with Maciej's central point, I think the inside arguments he presents are pretty weak. I think that AI risk is not a pressing concern even if you grant the AI risk crowd's assumptions. Elided from https://alexcbecker.net/blog.html#against-ai-risk: The real AI risk isn't an all-powerful savant which misinterprets a command to "make everyone on Earth happy" and destroys the Earth. It's a military AI that correctly interprets a command to kill a particular group of people, so effectively that its masters start thinking about the next group, and the next. It's smart factories that create a vast chasm between a new, tiny Hyperclass and the destitute masses... AI is hardly the only technology powerful enough to turn dangerous people into existential threats. We already have nuclear weapons, which like almost everything else are always getting cheaper to produce. Income inequality is already rising at a breathtaking pace. The internet has given birth to history's most powerful surveillance system and tools of propaganda.
以群体形式或印象来判断论证会误伤真正的异端突破，文章对人身攻击的背书有风险。
这篇文章明确为“人身攻击”式论证辩护：这些人穿古怪袍子、住在偏远营地、齐声说话很瘆人；就算他们的论证无可反驳，你的经验也告诉你这是邪教。问题在于，人身攻击并非从不奏效——邪教例子常能得出正确结论——但它失效的情形可能极其重要。99.9% 的 26 岁打零工、发明时间旅行理论的人是怪人；可如果你的规则是“看起来像怪人就全盘无视”，你就会错过狭义相对论（以及后来的广义相对论）。
apsec112查看英文原文
This article explicitly endorses argument ad hominem: "These people are wearing funny robes and beads, they live in a remote compound, and they speak in unison in a really creepy way. Even though their arguments are irrefutable, everything in your experience tells you you're dealing with a cult. Of course, they have a brilliant argument for why you should ignore those instincts, but that's the inside view talking. The outside view doesn't care about content, it sees the form and the context, and it doesn't look good." The problem with argument ad hominem isn't that it never works. It often leads to the correct conclusion, as in the cult case. But the cases where it doesn't work can be really, really important. 99.9% of 26-year-olds working random jobs inventing theories about time travel are cranks, but if the rule you use is "if they look like a crank, ignore everything they say", then you miss special relativity (and later general relativity).
最现实的 AI 风险是机器商业和金融工具在经济中自主优化，冲击人类。
真正让我恐惧的不是天网或 HAL，而是 Charles Stross《Accelerando》里写的：机器控制的商业——“狡猾的自我意识金融工具”——进化到理解和操纵经济的能力远超人类。可怕在于买卖商品和服务有实时、计算机化的效率激励：不只是股市，还有电商、智能合约、供需平台、按需服务、融资等；发现、质检、支付、合规等传统环节正迅速消失。计算机不能注册公司，但可通过 API 让人类名义董事代劳；公司早已是非人意图的理性构造。想象私募股权基金的智能合约依法强制企业按会计指标行动，一群智能投资机器人追踪这些机器人主导的基金——系统会多快优化，优化结果对地面上的人又意味着什么？在 AI 灾难场景里这最现实，因为有太多激励去推进它。
ian0查看英文原文
What really terrified me wasn't skynet or Hal, but what Charles Stross describes in his novel Accelerando. Basically, machine controlled businesses (“slyly self-aware financial instruments”) evolving to the point where their aptitude at understanding and manipulating economics eclipses anything a human can compete with. It terrified me because there is efficiency incentive for the buying/selling of goods and services to be realtime and computer based. Not just stock markets but also e-comm, smart contracts, supplier-buyer platforms, on-demand services, financing etc. The traditional infrastructure blocks (discovery, quality control, payments, compliance etc) are rapidly disappearing. There are some yet to be addressed, but they are not huge. For example a computer cannot incorporate a company. But it could easily interact via API to a human nominee who sets one up on their behalf. Companies are already non-human in intention but are a completely rational construct. And it doesnt even have to be anywhere near ground level. Imagine a smart contract from a private equity firm that legally forces businesses to act in a specific way based on accountancy metrics. And a bunch of smart investor bots rapidly tracking the progress of these bot-led funds. How quickly would the system optimise and what would those optimisations look like to people on the ground? Out of all the disaster situations with AI this one strikes me as so realistic because there is so much incentive to work towards it (as opposed to creating skynet, hal or the paperclip machine).
人类靠工具、策略和激励能控制更强实体，猫的类比反而削弱文章对人类无力的暗示。
这其实像是反向的完美论证。每天都有数百万人把猫放进箱子，尽管猫并不乐意。如果你出 1000 美元请一个正常、还算能干的人把不情愿的猫弄进箱子，你真觉得他完全做不到？人类还能把老虎关在动物园里，每年数百万人去看，严重受伤率极低——尽管老虎默认有攻击性，且能轻易压倒人。
apsec112查看英文原文
"Not many people know that Einstein was a burly, muscular fellow. But if Einstein tried to get a cat in a carrier, and the cat didn't want to go, you know what would happen to Einstein. He would have to resort to a brute-force solution that has nothing to do with intelligence, and in that matchup the cat could do pretty well for itself." This seems, actually, like a perfect argument going in the other direction. Every day, millions of people put cats into boxes, despite the cats not being interested. If you offered to pay a normal, reasonably competent person $1,000 to get a reluctant cat in a box, do you really think they simply would not be able to do it? Heck, humans manage to keep tigers in zoos, where millions of people see them every year, with a tiny serious injury rate, even though tigers are aggressive and predatory by default and can trivially overpower humans.
历次讨论
同类还值得读
如何写一个拼写纠正器
How to Write a Spelling Corrector
Peter Norvig 用一个几十行的 Python 程序演示如何做拼写纠正：先用编辑距离（删除、插入、替换、换位）生成所有可能的候选词，再用从大量文本中统计出的词频概率，选出…
AI 与机器学习上首页 10 次 · 2007–2022norvig.com编辑精选面向黑客的概率编程与贝叶斯方法
Probabilistic Programming and Bayesian Methods for Hackers (2013)
《黑客的贝叶斯方法与概率编程》是一本开源的贝叶斯统计入门书，以 Python/PyMC 代码和交互式 Jupyter Notebook 为主线，强调“先实践、后数学”的学习路径，…
AI 与机器学习上首页 5 次 · 2013–2018camdavidsonpilon.github.io编辑精选神经网络黑客指南
Hacker's Guide to Neural Networks
Andrej Karpathy 写的神经网络入门教程，面向程序员，从最简单的表达式求导讲起，逐步推导出反向传播和梯度下降，最后引出神经网络。
AI 与机器学习上首页 4 次 · 2014–2019karpathy.github.io编辑精选神经网络能逼近任意函数的可视化证明
A visual proof that neural nets can approximate any function
这一章来自 Michael Nielsen 的在线书《Neural Networks and Deep Learning》，用 sigmoid 单元叠加成局部隆起/平滑阶跃的图示…
AI 与机器学习上首页 4 次 · 2014–2022neuralnetworksanddeeplearning.com编辑精选论乔姆斯基与统计学习的两种文化
On Chomsky and the Two Cultures of Statistical Learning
Norvig 回应 Chomsky 对统计语言模型批评的文章，围绕“统计学习能否解释语言”展开，比较生成与解释、少参数模型与 Shannon 式高参数统计模型。
AI 与机器学习上首页 4 次 · 2011–2025norvig.com编辑精选