系统与基础设施dougseven.com编辑精选
Knightmare：一个 DevOps 警世故事（2014）
Knightmare: A DevOps Cautionary Tale (2014)
2014 — 2020，它一次次回到首页
黑点是上首页（≥ 50 分）的那次投稿，空心点是没冲上去的投稿，点越大分数越高。
导读
这篇文章复盘 Knight Capital 2012 年的著名事故：一次部署只更新了 8 台服务器中的 7 台，遗留的旧代码被触发，导致公司在约 45 分钟内损失约 4.4 亿美元，并最终被收购。作者借这起事故说明，在金融生产环境中，手动部署、缺少可验证流程以及缺少回滚/停机机制会带来怎样的代价。
为什么经久不衰：它是 DevOps 领域最经典的事故复盘之一：技术细节简单却致命，组织与流程问题至今仍在重演，提醒每个工程师敬畏生产发布。
HN 评论精选
用亲身经历说明运维命令的歧义和手速可能造成巨大风险，命名与操作习惯很重要。
我曾因为在 bash 里匆忙输入而关掉过一台算法交易服务器：先用 Ctrl-r 反向搜索历史，输入 'ps' 想找进程状态工具，然后按回车——结果发现 Ctrl-r 实际在历史里找到的是 'stopserver.sh'（stoPServer.sh 里也有个 ps）。5 秒内销售交易主管就打来电话，问为什么他的界面显示所有订单都被暂停了。幸运的是恢复代码很健壮，我大约半分钟就重启服务器并恢复交易。那是 2.5 亿到 4 亿美元订单被暂停半分钟，更别说我的心跳了。之后我把 stopserver.sh 改名为 stop_server.sh。P.S. 打字速度不只是被高估，在某些场景下还很危险。欲速则不达。
vijucat查看英文原文
I once shut down an algorithmic trading server by hastily typing (in bash): - Ctrl-r for reverse-search through history - typing 'ps' to find the process status utility (of course) - pressing Enter,....and realizing that Ctrl-r actually found 'stopserver.sh' in history instead. (There's a ps inside stoPServer.sh) I got a call from the head Sales Trader within 5 seconds asking why his GUI showed that all Orders were paused. Luckily, our recovery code was robust and I could restart the server and resume trading in half a minute or so. That's $250 million to $400 million of orders on pause for half a minute. Not to mention my heartbeat. Renamed stopserver.sh to stop_server.sh after that incident :| P.S. typing speed is not merely overrated, but dangerous in some contexts. Haste makes waste.
质疑为何缺少强制停机/硬重启机制，认为面对巨额损失应能快速切断系统。
每次读这个故事我都有一个不明白的问题：为什么他们不能直接关掉服务器？总该有某种机制才对。我的意思是，4 亿美元可不是小数目，不该犹豫到不用锤子砸服务器。看起来他们很早就发现问题，并在 45 分钟里至少花了一部分时间调试。我知道他们可能没有服务器的物理访问权限，但难道没有任何硬重启的办法吗？
NhanH查看英文原文
Everytime I'm reading the story, there is one question that I've never understood: why can't the just shutdown the servers itself? There ought to be some mechanism to do that. I mean, $400 millions is a lot of money to not just bash the server with a hammer. It seems like they realized the issue early on and was debugging for at least part of the 45 minutes. I know they might not have physical access to the server, but wouldn't there be any way to do a hard reboot?
追问事故背后的组织决策：为何自动化/可验证部署的建议可能被忽视，并用自身演练经历说明手动部署多步骤极易出错。
这类文章讲技术层面很有趣，但我总想知道组织/管理层面：Knight Capital 有没有人提出需要自动化、可验证的部署流程？如果提了，为什么被忽视？是否被视为浪费资源？自动化部署这么常见，很难相信没有工程师建议过。作者还讲了类似经历：部署大系统时，DBA 部分演练 20 多次没有一次无错，约 130 个步骤总有些被跳过；生产与非生产环境还有显著差异……
otakucode查看英文原文
While articles like this are very interesting for explaining the technical side of things, I am always left wondering about the organizational/managerial side of things. Had anyone at Knight Capital Group argued for the need of an automated and verifiable deployment process? If so, why were their concerns ignored? Was it seen as a worthless expenditure of resources? Given how common automated deployment is, I think it would be unlikely that none of the engineers involved ever recommended moving to a more automated system. I encountered something like this about a year ago at work. We were deploying an extremely large new system to replace a legacy one. The portion of the system which I work on required a great deal of DBA involvement for deployment. We, of course, practiced the deployment. We ran it more than 20 times against multiple different non-production environments. Not once in any of those attempts was the DBA portion of the deployment completed without error. There were around 130 steps involved and some of them would always get skipped. We also had the issue that the production environment contained some significant differences from the non-production environments (over the past decade we had, for example, delivered software fixes/enhancements which required database columns to be dropped... this was done on the non-production systems, but was not done on the production environment because dropping the columns would take a great deal of time). Myself and others tried to raise concerns about this, but in the end we were left to simply expect to do cleanup after problems were encountered. Luckily we were able to do the cleanup and the errors (of which there were a few) were able to be fixed in a timely manner. We also benefitted from other portions of the system having more severe issues, giving us some cover while we fixed up the new system. The result, however, could have been very bad. And since it wasn't, management is growing increasingly enamored with the idea of by-the-seat-of-your-pants development, hotfixes, etc. When it eventually bites us as I expect it will, I fear that no one will realize it was these practices that put us in danger.
提醒全自动部署本身也有风险，例如自动大规模下线实例，不能把自动化当成唯一答案。
我不认为文章的结论是‘唯一答案’，因为全自动部署流程还有另一种风险，AWS 和 Google 都曾在某个时候被它咬过：自动地大规模关停大量实例。
t0mas88查看英文原文
I'm not sure the conclusion of the post is the "One and Only Answer" because a fully automated deploy process has another risk that has bitten both AWS and Google at some point: fully automatically taking down huge amounts of instances.
历次讨论
同类还值得读
TTY 揭秘（2008）
The TTY demystified (2008)
经典文章深入剖析了 Unix/Linux 中 TTY 子系统的工作原理，包括线路规程、终端驱动、信号与控制字符等机制。
系统与基础设施上首页 11 次 · 2009–2025linusakesson.net编辑精选为什么 GNU grep 这么快
Why GNU grep is Fast
FreeBSD 邮件列表上一篇广为流传的帖子，解释 GNU grep 为什么快：核心在于 Boyer-Moore 字符串搜索算法配合精心优化的内部循环和 I/O，让程序在搜索时“…
系统与基础设施上首页 9 次 · 2010–2022lists.freebsd.orgXplain：为普通人讲解 X11
Xplain – Explaining X11 for the rest of us
《Xplain》是作者（HN 用户 Jasper_）写的一系列 X11 长文，用他在浏览器中重建的 X11 绘图模型做交互演示，讲解 X11 的绘图模型与窗口系统设计。
系统与基础设施上首页 6 次 · 2013–2020magcius.github.io编辑精选旋风教程：为 Linux 创建极小 ELF 可执行文件（1999）
A Whirlwind Tutorial on Creating Really Teensy ELF Executables for Linux (1999)
breadbox 写于 1999 年的经典教程，主题是如何为 Linux 手工构造并裁剪出体积极小的 ELF 可执行文件。
系统与基础设施上首页 6 次 · 2007–2025muppetlabs.comPostgreSQL 内部原理
The internals of PostgreSQL
《The Internals of PostgreSQL》是一本免费在线书籍，系统讲解 PostgreSQL 的内部实现，包括查询处理、并发控制、WAL、缓冲区管理等核心机制。
系统与基础设施上首页 5 次 · 2017–2025interdb.jp编辑精选