系统与基础设施github.com编辑精选
分布式计算的编程模型
Programming Models for Distributed Computation
2017 — 2020,它一次次回到首页
黑点是上首页(≥ 50 分)的那次投稿,空心点是没冲上去的投稿,点越大分数越高。
导读
这是 Heather Miller 编写的一本在线开源书稿的 GitHub 仓库,系统梳理分布式计算的各种编程模型——从 MapReduce、Dryad、Spark 到流处理与一致性模型,并附有配套课程的论文清单。书中按模型分类整理相关经典论文与设计思路,适合当作分布式计算领域的学习路线图。
为什么经久不衰:它把散落在论文里的分布式编程范式收拢成一份可读的脉络图,近十年过去仍是入门与回顾该领域的实用索引。
HN 评论精选
质疑把分布式与单机计算截然二分的框架,认为边界其实很模糊。
分布式环境里存在单机环境没有的问题:部分失败、并发、延迟,这三点让分布式计算与本地计算有本质区别。不过我认为这种框架化思维过度了——并发和延迟在单机系统里同样是问题,我们其实也希望单机系统能处理部分失败、能增量更新。把“分布式计算”当成与单机计算截然不同的领域,长期看不是好办法。归根结底,很少有系统真的是“单机”系统,机器都彼此连接、参与更大的系统。
shalabhc查看英文原文
> There are problems that exist in distributed system environments that do not exist in single-machine environments. Partial failure, concurrency, and latency are three problems that make distributed computing fundamentally different from local computing. IMO this can frame our thinking too much. Concurrency and latency are problems in single systems as well. It would be great to have single systems that handle partial failure, can be updated incrementally, etc. Thinking of 'distributed computing' as a different domain than single system computing is not a useful long term approach. In the end very few systems are 'single machine' systems, if at all - machines are all connected to each other and participate in other, larger, systems.
指出书稿缺少 Spanner/Calvin 等强一致性数据库算法的内容。
这看起来是一份相当完整的综述,但漏掉了一个重要主题:Spanner、FaunaDB 这类数据库所用的强一致性算法。这两个全球一致性数据库用的是不同族系的算法,两族都值得收录。
jchanimal查看英文原文
This looks like a reasonably complete survey. However, one important topic is missing. That is the strong consistency algorithms used by databases like Spanner and FaunaDB. Those two globally consistent databases use different families of algorithms, so it’s probably worth including both. Here is a comparison between them: https://fauna.com/blog/distributed-consistency-at-scale-spanner-vs-calvin
业内研究者指出书稿对 DryadLINQ、Naiad 等工作的定位有失公允。
我心情有点矛盾:我很高兴有人读论文并写综述,但我觉得该改的地方太多,又不想直接闯进去提 PR。比如它基本忽略了 DryadLINQ——我认为它是“批处理计算”领域仅次于 MapReduce 的第二重要论文,书中却只把它当作写 Dryad 程序时该用的东西,理由是 Dryad 的编程模型“不如 MapReduce 优雅”,评价太苛刻。流处理章节绕来绕去,然后暗示 Google Dataflow 统一了先前的工作,可你去读 Dataflow 论文搜一下“Naiad”,人家根本没提。这些批评似乎过于轻信其他论文相关工作章节的说法。
frankmcsherry查看英文原文
I am a bit torn with this, because while I love people reading and then writing up things, there are so many changes I think they should make but I don't want to just barge in and make pull requests. E.g., they largely miss DryadLINQ (disclosure: not an author, but worked with them), which I would rate as the second most important paper after MapReduce in the "batch compute" space. It shows up only as what you should use to write Dryad programs, because Dryad's programming model "is less elegant than MapReduce"; harsh. The streaming section (disclosure: Naiad co-author) dances around and then suggests that Google Dataflow is somehow a unification of previous work, which is .. well, you can read the Google Dataflow paper http://www.vldb.org/pvldb/vol8/p1792-Akidau.pdf and do a search for "Naiad"; it's not even on their radar. It feels a bit like the critiques are perhaps too-credulous readings of the related work sections of other papers. I'm not sure a great way to get to a consensus understanding of the position of the works in the historical record, or whether it is even possible with sensitive authors (hi), but it would be cool to have. Any recommendations would be great.
提示可直接阅读排版好的 HTML 版书稿。
给没发现的人:不用看原始 markdown,这本书已有浏览器可直接渲染的 HTML 版本:http://dist-prog-book.com/chapter/
zzzcpan查看英文原文
For those who didn't figure it out - instead of raw markdown you can read the book already in html format that browsers can render: http://dist-prog-book.com/chapter/
历次讨论
同类还值得读
TTY 揭秘(2008)
The TTY demystified (2008)
经典文章深入剖析了 Unix/Linux 中 TTY 子系统的工作原理,包括线路规程、终端驱动、信号与控制字符等机制。
系统与基础设施上首页 11 次 · 2009–2025linusakesson.net编辑精选为什么 GNU grep 这么快
Why GNU grep is Fast
FreeBSD 邮件列表上一篇广为流传的帖子,解释 GNU grep 为什么快:核心在于 Boyer-Moore 字符串搜索算法配合精心优化的内部循环和 I/O,让程序在搜索时“…
系统与基础设施上首页 9 次 · 2010–2022lists.freebsd.orgXplain:为普通人讲解 X11
Xplain – Explaining X11 for the rest of us
《Xplain》是作者(HN 用户 Jasper_)写的一系列 X11 长文,用他在浏览器中重建的 X11 绘图模型做交互演示,讲解 X11 的绘图模型与窗口系统设计。
系统与基础设施上首页 6 次 · 2013–2020magcius.github.io编辑精选旋风教程:为 Linux 创建极小 ELF 可执行文件(1999)
A Whirlwind Tutorial on Creating Really Teensy ELF Executables for Linux (1999)
breadbox 写于 1999 年的经典教程,主题是如何为 Linux 手工构造并裁剪出体积极小的 ELF 可执行文件。
系统与基础设施上首页 6 次 · 2007–2025muppetlabs.comPostgreSQL 内部原理
The internals of PostgreSQL
《The Internals of PostgreSQL》是一本免费在线书籍,系统讲解 PostgreSQL 的内部实现,包括查询处理、并发控制、WAL、缓冲区管理等核心机制。
系统与基础设施上首页 5 次 · 2017–2025interdb.jp编辑精选