系统与基础设施gallium.inria.fr
我在 Intel Skylake 处理器中发现了一个 bug
I found a bug in Intel Skylake processors
2017 — 2021，它一次次回到首页
黑点是上首页（≥ 50 分）的那次投稿，空心点是没冲上去的投稿，点越大分数越高。
导读
这是 INRIA Gallium 团队的一篇著名排障记：他们发现某个 OCaml 程序只在特定 GCC 优化级别下、于 Intel Skylake 机器上偶发崩溃，经过长时间排查（包括反复用 GDB 复现、跑上千次抓 core dump），最终把故障缩小到 Skylake 的一条硬件勘误（SKL150，与短循环中使用 AH/BH/CH/DH 寄存器有关），并推动与 Intel 的沟通。文中详细记录了从应用层一路怀疑到 CPU 微架构的推理过程。
为什么经久不衰：它把"软件 bug 还是硬件 bug"这条最难走的排查路径完整展示出来，既有方法论价值，也提醒人们：底层硬件勘误会以完全无法从汇编看出的方式咬到上层程序。
HN 评论精选
用户忽视了顶级专家的免费调试建议，颇为讽刺。
这个故事的搞笑之处在于：那位"严肃的工业 OCaml 用户"无视了 Xavier Leroy 给出的直接、相对容易实施且非常可靠的调试建议。我但愿自己在那种处境下，被这种级别的专家指点时，至少会照着他的建议试试。不用专家的建议，那找专家干嘛？这让人想到那些不听医生劝告的人。
piemonkey查看英文原文
One amusing thing about this epic tale is Serious Industrial OCaml User disregarded direct, relatively easy to implement, very sound advice from Xavier Leroy about how to debug their system! I would like to think that, were I in a similar situation being advised by an expert of that calibre, I would at least humor his suggestions. Why seek the expert if not for his advice? It brings to mind people disregarding doctors who give them inconvenient medical advice.
从微架构角度推测微码补丁可能是禁用 Loop Stream Detector，并追问性能代价。
comp.arch 上有人指出：勘误说的是问题出现在少于 64 条指令、使用 AH/BH/CH/DH 的短循环中。而 Skylake 的指令解码队列是每线程 128 uOps（双线程时 2×64），Loop Stream Detector 每线程最多检测 64 uOps 的循环。所以那个微码更新也许只是把 loopback detector 关掉了。如果 bug 出在循环检测器里、补丁只是禁用它而非修复，那有人有前后性能对比数据吗？
willvarfar查看英文原文
A comp.arch poster said: > The errata refers to the problem showing up on short loops of less than 64 instructions that use AH, BH, CH or DH. > Looking at the Skylake microarch, the instruction decode queue is 128 uOps thread, 2*64 uOps when threaded. The Loop Stream Detector "can stream the same sequence of µOPs directly from the IDQ continuously without any additional fetching, decoding, or utilizing additional caches or resources." ... "capable of detecting loops up to 64 µOPs per thread". https://en.wikichip.org/wiki/intel/microarchitectures/skylake#.C2.B5OP-Fusion_.26_LSD > So maybe the microcode update just shuts off the loopback detector. https://groups.google.com/d/msg/comp.arch/UkO4Z2FT18c/7YlC0aH7AQAJ So if the bug is in the loop-detector, and the patch possibly disables it rather than fixes it, then does anyone have any before-and-after performance stats?
推荐用 rr 做确定性重放，比反复抓 core dump 更高效。
作者是先拿 SIOU（那位 OCaml 用户）提供的可执行文件，在 GDB 下交互调试（有时要等一小时才复现一次崩溃），然后写了个小 OCaml 脚本跑 1000 次并保存每次崩溃的 core dump。rr 在这种情况下常常更省时，它能提供直到崩溃点的确定性重放，而 core dump 分析只是事后的一次性快照。
ihnorton查看英文原文
> I worked from the executable provided by SIOU, first interactively under GDB (but it nearly drove me crazy, as I had to wait sometimes one hour to trigger the crash again), then using a little OCaml script that ran the program 1000 times and saved the core dumps produced at every crash. rr can often be a time-saver in situations by providing deterministic replays up to the point of a crash, whereas coredump analysis is a single retrospective snapshot. http://rr-project.org/
先怀疑自己代码是最合理的第一直觉，但遇到深层 bug 时代价巨大。
栈深处 bug 的麻烦在于：要确认它真的那么深，非常耗时。我曾写过一个 Swift iOS 应用，遇到某个集合类的问题。第一反应当然不会怀疑 Swift 标准库，于是一个个假设排查自己的代码，慢慢删掉可疑片段和组合，还减少条目数以简化日志——结果发现是集合大小超过 16 才出问题，而这根本不是个假设，只是偶然缩减 n 才撞上。我还发现 release 正常、debug 异常，一度以为是竞态。最后我干脆新建工程专门测那个集合类，纯粹为了周全而非真觉得库有 bug，结果一复现发到 StackOverflow 上，苹果的人就确认了并修好了。直接去测库本可以省下大量时间，但"先怀疑自己的代码"仍是最合理的第一启发式，这就是代价。
lordnacho查看英文原文
The problem with bugs deep in the stack is that it is really time consuming to establish that they are in fact as deep as they are. I wrote a Swift iOS app once, and came across an issue with one of the collection classes. Of course, nobody thinks that the Swift libs will be wrong as a first guess. So I worked through a number of hypotheses about my own code, slowly stripping out pieces that I thought might contain an error. And then combinations. I also tried reducing the number of entries just to simplify the logs. This worked, but of course you are not going to think that there's a library bug affecting collections with size > 16, and it wasn't actually a theory until I randomly decided to reduce the n. I also discovered that it worked just fine on release but not debug, so I thought maybe I have some race condition. More and more stripping down occurred, until I eventually gave up using my own project and just started a new one just to see about the collection class. I did it for the sake of being thorough, rather than actually thinking the lib had a bug in its debug implementation. But lo and behold, when I managed to make it reproducible and put it on SO, someone from Apple acknowledged that they could also see it, and they fixed it. Naturally if I'd gone direct to testing the lib I'd have saved a huge amount of time, but I guess that's the tradeoff from the most sensible heuristic: test your own code first, the bug is there.
历次讨论
同类还值得读
TTY 揭秘（2008）
The TTY demystified (2008)
经典文章深入剖析了 Unix/Linux 中 TTY 子系统的工作原理，包括线路规程、终端驱动、信号与控制字符等机制。
系统与基础设施上首页 11 次 · 2009–2025linusakesson.net编辑精选为什么 GNU grep 这么快
Why GNU grep is Fast
FreeBSD 邮件列表上一篇广为流传的帖子，解释 GNU grep 为什么快：核心在于 Boyer-Moore 字符串搜索算法配合精心优化的内部循环和 I/O，让程序在搜索时“…
系统与基础设施上首页 9 次 · 2010–2022lists.freebsd.orgXplain：为普通人讲解 X11
Xplain – Explaining X11 for the rest of us
《Xplain》是作者（HN 用户 Jasper_）写的一系列 X11 长文，用他在浏览器中重建的 X11 绘图模型做交互演示，讲解 X11 的绘图模型与窗口系统设计。
系统与基础设施上首页 6 次 · 2013–2020magcius.github.io编辑精选旋风教程：为 Linux 创建极小 ELF 可执行文件（1999）
A Whirlwind Tutorial on Creating Really Teensy ELF Executables for Linux (1999)
breadbox 写于 1999 年的经典教程，主题是如何为 Linux 手工构造并裁剪出体积极小的 ELF 可执行文件。
系统与基础设施上首页 6 次 · 2007–2025muppetlabs.comPostgreSQL 内部原理
The internals of PostgreSQL
《The Internals of PostgreSQL》是一本免费在线书籍，系统讲解 PostgreSQL 的内部实现，包括查询处理、并发控制、WAL、缓冲区管理等核心机制。
系统与基础设施上首页 5 次 · 2017–2025interdb.jp编辑精选