系统与基础设施somethingsimilar.com编辑精选
给分布式系统新人的笔记（2013）
Notes on Distributed Systems for Young Bloods (2013)
2013 — 2024，它一次次回到首页
黑点是上首页（≥ 50 分）的那次投稿，空心点是没冲上去的投稿，点越大分数越高。
导读
这是一篇面向分布式系统新手的经验笔记，作者以格言式的短条总结了多年实战中踩过的坑，涵盖一致性、延迟、背压、监控、CAP 定理等主题。文章多次登上 Hacker News 首页，被视为分布式系统入门的经典读物。
为什么经久不衰：它用简练、可操作的经验法则浓缩了分布式系统的核心陷阱，历经十余年仍被反复推荐，是新手建立直觉的捷径。
HN 评论精选
分布式容错早在 80 年代的 Tandem 系统中就已在 OS 和数据库层解决，如今在应用层处理是倒退
在应用层处理分布式是一种倒退。Tandem Computers 在上世纪 80 年代就在操作系统和数据库层面把分布式做得很好，技术至今仍可从 HP 获得。Tandem 的 NonStop OS 与 UNIX 类系统有本质不同：UNIX 是带文件系统的分时系统，NonStop 是带数据库的事务系统，数据库之外几乎没有状态。程序启动、执行一个事务、退出；事务出问题就整体中止、不提交。Tandem 装机系统的平均运行时间约 10 年，主要故障原因是硬件维护时的人为失误。他们的硬件有大量自检，目标是检测错误并中止而非修复，事务失败会切换到配对的备份机并重试恢复。主要问题是硬件成本高，而且由于一系列收购，系统从 Tandem 移植到 MIPS、Alpha、HP-RISC、Itanium，最后到 x86——x86 缺少校验硬件，不再适合这种方案。
Animats查看英文原文
It's a step backwards that distribution is being dealt with at the application level. Tandem Computers [1] had distribution at the OS and database level working well in the 1980s. The technology is still available from HP. Tandem's NonStop OS is fundamentally different from UNIX-like OSs. UNIX is a time-sharing system with a file system. NonStop is a transaction system with a database system. There's very little state outside the database. Programs are started, do one transaction, and exit. If there's a problem during a transaction, the entire transaction aborts and does not commit. Tandem was running an average system uptime of about 10 years across their installed base, with the main cause of failure being human errors during hardware maintenance. One of their people wrote a paper on how to get the uptime up to 50 years. Tandem's own hardware had heavy self-checking. The goal was to detect errors and abort, not fix them. Failing a transaction would result in a fallover to the paired backup machine, which would result in a retry and recovery. Tandem's main problem was high hardware cost; you needed several of anything. Also, because of a series of acquisitions, the system had to be ported from Tandem to MIPS to Alpha to HP-RISC to Itanium to, finally, x86. X86 lacks the checking hardware, so it's not as good for this. [1] https://en.wikipedia.org/wiki/Tandem_Computers
没有模型检查或形式化证明，实现必然有 bug；“久经考验”不等于可靠
你可能以为自己理解了规格说明，但除非用过模型检查器或证明助手，实现里一定有 bug。单元测试不是规格（顶多是不完整的规格）。要习惯谓词演算、时序逻辑和写证明；如果你不能用可验证的模型或证明表达设计，就是在故意给系统留 bug。这些 bug 可能埋在 40 多层深的代码路径里，要各种条件恰好凑齐才触发。在 Gustafson 定律使权衡变得可接受的规模下，这类错误就不再是“极不可能”了。别把“久经考验”当作可靠系统或算法的标志——那只说明有人愿意承担风险并为此付出了几年代价。系统里仍有 bug，它们会影响你或你的用户（往往两者都影响）。
agentultra查看英文原文
You may think you understand your specification but unless you used a model checker or proof assistant there will be bugs in your implementation. Unit tests are not a specification (or are, at best, incomplete). Get used to predicate calculus, temporal logic, and writing proofs. If you cannot express a design with a model or proof that can be verified you're purposefully leaving bugs in your system. They may be buried > 40 steps deep in some code path where the stars have to align just right to trigger it. At the scale where Gustafson's law makes the trade-offs acceptable you will find that such errors are no longer, "highly improbable." Don't accept, "battle-tested," as the hallmark of a reliable system or algorithm -- all that means is that people willing to tolerate the risk did so and paid for it for a few years. There are still bugs in that system and they will affect you or your users (and often both).
背压常被忽视却至关重要；作者用 Erlang 邮箱爆掉的亲身经历说明要在生产负载下测试
“在整个系统中实现背压”这条很微妙，常被 CAP 定理的讨论淹没，但非常重要。我最近就在为加背压挣扎：有些地方用发消息的方式（Erlang 的 gen_server:cast），发送方把 30 多万条消息塞进接收方邮箱，接收方因瓶颈和 bug 处理不过来，内存涨到 120GB 后最终重启。简单的解法就是把 cast 换成 call，即让接收方确认、发送方等确认再发下一条（顺带说，这正是 Erlang/Elixir 的好处，只改了两行；如果是序列化库加 RPC socket 层，要写多得多的代码）。由此得出的推论是：一定要在生产负载和条件下测试，而不只是在笔记本上。本地用小测试数据时能装进内存，10GB 左右时问题不明显，所以看起来一切正常。
rdtsc查看英文原文
> Implement backpressure throughout your system. This is a subtle one and gets lost among CAP theorem talk, but it is important. Just recently had to struggle with adding backpressure. Did some things as just sending a message (a "gen_server:cast" in Erlang). The sender shoved all 300K+ messages in the mailbox of receiver which could not process them in time because of a bottleneck and bug. Receiver's memory usage went to 120GB when it eventually restarted. The simple solution was just to switch a gen_server:cast with a gen_server:call i.e. make the receiver acknowledge, and the sender wait for acknowledge before sending another message. (As an aside that's the nice benefit of Erlang/Elixir there, it was really a 2 line change. If it was some kind of serialization library + rpc socket layer, it would have been a lot more code to write). An obvious corollary of this is "always test on production workload and conditions, not just your laptop". What had happened locally is with a smaller test data it would fit in memory and for a 10Gb or so it wasn't noticeable. So it seemed like everything was working fine. But speaking of CAP theorem this is always a great fun: http://ferd.ca/beating-the-cap-theorem-checklist.html --- Beating the CAP Theorem Checklist Your ( ) tweet ( ) blog post ( ) marketing material ( ) online comment advocates a way to beat the CAP theorem. Your idea will not work. Here is why it won't work: ... ---
补充经验：事务尽量单机、控制载荷大小、为可诊断性设计、警惕时序问题
我想往清单里再加几条：如果有事务，尽量把事务限制在单机/单实例/单数据库内，跨机或软件事务是最后才该尝试的方案；注意载荷大小，别接近把网络打满，否则会出现诡异的“应用变慢”问题；为生产系统的可测试性和可诊断性而设计，如果是 Java，到处都用 JMX 扩展；时间（和计时）是你的敌人，要把它变成朋友。
critium查看英文原文
I would add to the list; * If you have transactions, keep your transactions to a single machine/instance/db as much as possible. Muti-machine or software transactions are the LAST solution you should try. * Pay attention to payload sizes. Make sure you dont come close to saturating the network. Which leads to weird "app is slow" problems. * Design for testability and diagnosability of production systems. If this is java, use JMX extensions EVERYWHERE for EVERYTHING. * Time (and timing) is your enemy. Make it your friend. EDIT: Side note. IMO, JMX extensions are one of the most under-appreciated things that java and jvm devs have but keep forgetting about but its so powerful.
历次讨论
同类还值得读
TTY 揭秘（2008）
The TTY demystified (2008)
经典文章深入剖析了 Unix/Linux 中 TTY 子系统的工作原理，包括线路规程、终端驱动、信号与控制字符等机制。
系统与基础设施上首页 11 次 · 2009–2025linusakesson.net编辑精选为什么 GNU grep 这么快
Why GNU grep is Fast
FreeBSD 邮件列表上一篇广为流传的帖子，解释 GNU grep 为什么快：核心在于 Boyer-Moore 字符串搜索算法配合精心优化的内部循环和 I/O，让程序在搜索时“…
系统与基础设施上首页 9 次 · 2010–2022lists.freebsd.orgXplain：为普通人讲解 X11
Xplain – Explaining X11 for the rest of us
《Xplain》是作者（HN 用户 Jasper_）写的一系列 X11 长文，用他在浏览器中重建的 X11 绘图模型做交互演示，讲解 X11 的绘图模型与窗口系统设计。
系统与基础设施上首页 6 次 · 2013–2020magcius.github.io编辑精选旋风教程：为 Linux 创建极小 ELF 可执行文件（1999）
A Whirlwind Tutorial on Creating Really Teensy ELF Executables for Linux (1999)
breadbox 写于 1999 年的经典教程，主题是如何为 Linux 手工构造并裁剪出体积极小的 ELF 可执行文件。
系统与基础设施上首页 6 次 · 2007–2025muppetlabs.comPostgreSQL 内部原理
The internals of PostgreSQL
《The Internals of PostgreSQL》是一本免费在线书籍，系统讲解 PostgreSQL 的内部实现，包括查询处理、并发控制、WAL、缓冲区管理等核心机制。
系统与基础设施上首页 5 次 · 2017–2025interdb.jp编辑精选