编程与软件工程gamasutra.com
我遇过最难的 Bug
My Hardest Bug Ever
2013 — 2017，它一次次回到首页
黑点是上首页（≥ 50 分）的那次投稿，空心点是没冲上去的投稿，点越大分数越高。
导读
游戏开发者 Dave Baggett（后为 ITA Software 联合创始人）讲述他职业生涯中最难调试的一个 bug：在开发《Crash Bandicoot》时，游戏偶尔在特定关卡莫名崩溃，最终发现是索尼官方驱动库中一个函数在特定内存地址下会破坏数据——问题出在硬件手册未记载的底层行为上。文章详细还原了他数月排查、层层排除假设的全过程。
为什么经久不衰：它把“最难 bug 往往出在你最不敢怀疑的地方（编译器、硬件、官方库）”这一普遍经验讲成了一个引人入胜的侦探故事，任何工程师读来都会心一笑。
HN 评论精选
每个嵌入式工程师都会经历“那一刻”：无法复现的 bug 最终靠偏执的防御性代码“封印”。
在硬件/嵌入式领域，这被称为“那一刻”（The Moment）。项目进展顺利，一切比平时还顺畅——暴风雨前的平静。突然设备毫无征兆地失效了。你起初不当回事，它又出现了；你花几个小时想稳定复现，却怎么也做不到。于是你开始给代码“加钢板”：到处写偏执的错误检查、变量赋值两遍、重置、给时序参数加余量。代码看起来像疯子写的，但 bug 再也没出现。从那以后，只要项目哪里感觉不对，你就会往代码里插入近乎仪式性的咒语。
fpgaminer查看英文原文
In the hardware/embedded engineering world, this is what is known as The Moment. How long it takes for a developer to experience The Moment is correlated with how closely they are to the metal. I'm not sure if it's the solder, the silicon, or the flux, but it seeps into your mind and slowly, but surely, The Moment will happen. The project is going swimingly well. Smoother than normal, actually. The calm before the storm. Suddenly, abruptly, your device fails. Maybe you were there, maybe someone else reported it; neither makes it less mysterious. You brush it off with more important things to do. It happens again. Now it bothers you; top priority. You try for hours to make it happen consistently. No amount of tin foil, bunny ears, or interpretive dance will make it happen again. So you give up, you start steel plating the code. Paranoid error checking everywhere, setting variables twice, resetting, extra padding on all the timing parameters. PARANOIA. You code begins to look like a lunatic wrote it. But the bug never happens again. Your paranoia and double guessing fixed the problem. The Moment! From then on, whenever something doesn't quite _feel_ right with your projects, you start inserting strange, almost ritualistic incantations in your code. If anyone asks why you decided to sleep for 14.23ms instead of the prescribed 10ms, you just call them a devil and run away back to your ancient cross compiling GCC and make a sacrifice to the Upstream Gods. I feel for the author. From that Moment on, his code will never be the same. His sanity, now left on the cutting room floor along with the trust he once had in Datasheets and Programming Manuals. Embedded 4 Life.
在嵌入式领域，怀疑硬件和编译器的门槛比一般认知低得多。
“作为程序员，你学会第一、第二、第三次先怪自己的代码……大约到第一万次才怪编译器，再往后才怪硬件。”——作为嵌入式程序员，经常面对新硬件和自研编译器，怪硬件排到第五位、编译器第六位就够了。
AceJohnny2查看英文原文
"As a programmer, you learn to blame your code first, second, and third... and somewhere around 10,000th you blame the compiler. Well down the list after that, you blame the hardware." I dunno, as an embedded programmer working on new HW and sometimes with a custom compiler, blaming the HW became the 5th thing, and the compiler the 6th...
作者背景不凡：他后来创办的 ITA Software 解决了交通定价这类“分形级”复杂问题。
Dave 也是 ITA Software 的联合创始人，这家公司可以说是现代交通行业的引擎，驱动着几乎所有交通类网站（Hipmunk、Kayak 等）。用他的话说，这是个“分形问题”——从十万英尺高空看只是一个小点，越靠近复杂度越难以想象。这类问题（邮件、交通、电商、税务等）解决起来价值巨大，但需要海量工作。
josh2600查看英文原文
Just in case you guys don't know this, since I don't see it mentioned anywhere, Dave is also the co-founder of ITA software which is basically the engine holding up modern transportation. It powers almost all of the websites that deal in transportation (hipmunk, kayak, etc.), and this is, and I believe this is Dave's term, a fractal problem. Fractal problems are those that look really simple from 100,000 feet up (like a dot) and reach unimaginable complexity as you get closer (like a fractal). There are a lot of problems of this nature (like email, transportation, ecommerce, taxation, etc.) and solving them is worth a lot of money but requires a LOT of work.
老主机的硬件怪癖常被开发者当特性利用，如今模拟器还得忠实复现它们。
这种硬件怪癖在老游戏主机上非常常见。NES 和 SNES 时代，这类问题不仅靠圈内口口相传，还有一半被当作“特性”来利用，把硬件性能榨到极致。多年后模拟器必须忠实复现这些怪癖，否则游戏无法正确运行。
JoshTriplett查看英文原文
This kind of hardware quirk is really common in old consoles. In the NES and SNES era, not only were these kinds of issues passed around by tribal knowledge, but half of them were repurposed as features and used to push the hardware further. Not random save corruption issues, obviously, but many other hardware quirks became semi-documented "features" of the console, and years later, emulators would have to reproduce them faithfully or games wouldn't run correctly.
一线工程师分享多个硬件级诡异 bug：只写寄存器、温度敏感芯片、坏缓存，教训是端到端校验。
类似的经历：只写寄存器是万恶之源，两个写操作之间来了中断导致调度器停摆，排查两周、修复只需交换两条指令；USB 控制器有温度问题，早上和午饭后才正常，最初用一袋冰块压在芯片上找到了线索；磁盘缓存坏位会损坏文件，教训是：I/O 系统不可靠时再多 ECC 也没用，真正在乎的数据要做端到端校验（我们用 Merkle 树抓到了这个 bug）。
kabdib查看英文原文
Ah yes: - An OS would write the "please wake me up on the next interrupt" to a hardware register, then a shadow RAM location. Interrupt happened between the two writes, so the task scheduler wrote the wrong value to the register and stopped. Finding the problem: Two weeks. Fixing the problem: Swapped two instructions and rebuilt. Write-only registers are evil badness. - USB controller with a temperature problem: It would also work first thing in the morning, and after lunch. After maybe four days of correlating this the "Aha!" was visible on my face at lunch and I left the table rather abruptly. The initial fix was a bag of ice on the chip. Real fix took several weeks of increasingly loud conversations with the chip vendor. - Cache on a disk drive had a bad bit, occasionally corrupting file contents or (even more fun) directory structures. Memo: All the ECC in the world won't help you if your I/O system is flappin' in the breeze. End-to-end checksum anything you truly care about (in fact, we caught this one with a Merkle tree). Never mind the heap corruptions by code you don't even have source for, the async callbacks that fire long after you've nuked the objects and assumed everything was dead, the DMAs that came from outer space because you set them up hours ago and they finally fired, wiping out something random every time, the incoherent cache coherency systems, the serial ports that drop bytes if you send too fast . . . it's amazing anything sufficiently complex actually works.
历次讨论
同类还值得读
开源应用架构
The Architecture of Open Source Applications
《开源软件架构》（AOSA）是一本在线免费书，邀请知名开源项目的作者亲自剖析其内部设计，涵盖 SQLAlchemy、LLVM、Audacity、Apache、Bash 等系统。
编程与软件工程上首页 9 次 · 2011–2024aosabook.org编辑精选编程语言简史：简短、不完整且多半错误
A Brief, Incomplete, and Mostly Wrong History of Programming Languages
一篇以戏谑、故意不准确的方式回顾编程语言史的博客文章，用夸张轶事和讽刺定义串起 Lisp、C、C++、Java、JavaScript、PHP 等语言。
编程与软件工程上首页 8 次 · 2009–2021james-iry.blogspot.comSQLite 是如何测试的
How SQLite is tested
SQLite 官方文档页，系统介绍 SQLite 的测试体系：多种测试方法（TCL 测试、模糊测试、回归测试、静态分析等）和海量测试用例，展示了这个小型开源项目如何靠极致测试获得…
编程与软件工程上首页 8 次 · 2009–2025sqlite.org编辑精选高级编程语言（2009）
Advanced programming languages (2009)
Matt Might 的经典文章，按用途推荐“高级编程语言”：如果追求就业就选主流语言，如果追求表达力、学术研究或创业，则推荐 Haskell、OCaml、Scala 等函数式语…
编程与软件工程上首页 8 次 · 2009–2022matt.might.net我的 Forth 与栈机器经历
Yosefk - My history with Forth & stack machines
作者以亲身经历回顾了使用 Forth 语言和栈式机器的历程：既赞叹其简洁优雅、代码量小、可自举的优点，也坦率指出栈操作在实际编程中的痛苦，以及 Forth 社区文化中的问题。
编程与软件工程上首页 8 次 · 2010–2024yosefk.com编辑精选