系统与基础设施blog.discordapp.com
Discord 如何将 Elixir 扩展到 500 万并发用户
How Discord Scaled Elixir to 5M Concurrent Users
2017 — 2019，它一次次回到首页
黑点是上首页（≥ 50 分）的那次投稿，空心点是没冲上去的投稿，点越大分数越高。
导读
Discord 工程团队分享他们如何用 Elixir/Erlang VM 将实时消息系统扩展到 500 万并发用户，包括利用 mochiglobal 共享只读数据、ets:update_counter 等优化手段。文章在 2017 和 2019 两次登上 HN，2019 年 Discord 工程师还在评论区补充了后续进展：并发规模已远超 500 万，并用 Rust 加速 Elixir、开源了 ZenMonitor 等组件。
为什么经久不衰：它是少有的来自大规模生产环境的第一手 Elixir 扩展性实践记录，且作者本人在多年后回帖更新进展，对理解 BEAM 生态的真实能力仍有很高参考价值。
HN 评论精选
原作者团队现身说法：系统规模已远超文中数字，并持续用 Rust 和新工具优化。
Discord 基础设施工程师在此——这篇博客需要更新了！此后我们把这套系统扩展了很多：Fortnite 官方服务器并发用户已超过 10 万，Discord 整体也远超 500 万并发；我们在某些地方用 Rust 让 Elixir 跑得更快，还构建了一个可大规模扩展的 Process.monitor 通用替代品，下周将在 Code BEAM SF 开源。核心基础设施只有 4 名全职工程师，全公司约 40 名工程师，欢迎联系我。
xb95查看英文原文
Discord infra engineer here -- this blog post needs an update! Since then we've scaled this system much more. :) The Fortnite Official server has exceeded 100,000 concurrent users, Discord itself is way past that 5M concurrent number, we're now using Rust in certain places to make Elixir go faster, we've built a general purpose replacement to Process.monitor that scales a whole truckload more that we're open sourcing next week at Code BEAM SF... the list goes on. There's a lot of fun stuff going on to try to make this system even more efficient and reliable, there's a lot to do still. We run everything on a very small engineering team (there are 4 fulltime engineers on the core infrastructure, only about 40 engineers in the whole company) and we're always looking for a few more. Feel free to reach out to me (zorkian#0001 on Discord) if this blog post sounds up your alley!
补充技术细节：fastglobal 因重编译成本被弃用，改用 ETS 优化，故障恢复能力大幅提升。
我们后来弃用了 fastglobal，因为随着 BEAM 节点上进程数增长，重新编译代码耗时太长——重编译成本与节点进程数线性相关，达到每节点百万进程后实在太慢太耗资源。我们后来从 ETS 中榨出了所需性能来运行 hashring。另外多亏了 ZenMonitor，现在单个 guilds 节点故障约 40 秒即可恢复，上周五有个节点因宿主机重启而故障，系统自动恢复后值班人员才收到事后告警——放在以前这会引发全系统级联故障。一个 guild 节点承载 60-70 万个并发服务器，整个分布式系统完整重启约需 17 分钟。
jhgg查看英文原文
We've since ditched fastglobal actually, as we found that it'd take too long to recompile code as the number of processes on the BEAM VM grew. As it turns out, the cost to recompile is linear scaling up with the # of processes on the node. Once we reached the million processes per node threshold it ended up being too slow, and resource intensive to do fastglobal dynamic module recompilation. We've since been able to squeeze out the performance from ETS we needed to be able to run our hashring. More on that in this PR here: https://github.com/discordapp/ex_hash_ring/pull/1 Also one amazing feat we've accomplished thanks to ZenMonitor (which we will open source soon as my colleague mentioned), is that we can now tolerate a guilds node failure and recover from it in ~40 seconds. We had a node fail on Friday actually due to the underlying host rebooting (gcp calls this a 'host error'), and didn't even notice until after the system had already recovered and the on-call got alerted after the fact. Back in the day this lead to a cascading failure throughout the system. A guild node runs between 600k-700k concurrent discord guilds, or servers as it's known in userland. And although we haven't done it recently, we clock a full restart of our distributed system at roughly 17 minutes (from shutdown to service fully restored).
Elixir 上手简单且背靠 Erlang VM，规模化时能避免昂贵重写，但采用它仍需组织层面的勇气。
这篇文章让我更加确信 Elixir 会成为大规模应用扩展领域的重要玩家。我最喜欢 Elixir 的一点是上手容易，同时站在 Erlang VM 这个巨人肩膀上：可以从简单应用起步，一旦规模变大，有大量久经考验的工具帮你省去头疼和昂贵的重写。当然今天用 Elixir 仍是一场豪赌，需要说服同事和老板承担风险，但可以放心它在需要更上一层楼时不会让你失望。没有什么东西是免费的，Erlang VM 也不是银弹，仍需团队投入精力调优，但深挖之后你会发现以更低成本解决问题的方法。
iagooar查看英文原文
This writeup make me even more convinced of Elixir becoming one of the large players when it comes to hugely scaling applications. If there is one thing I truly love about Elixir, it is the easiness of getting started, while standing on the shoulders of a giant that is the Erlang VM. You can start by building a simple, not very demanding application with it, yet once you hit a large scale, there is plenty of battle-proven tools to save you massive headaches and costly rewrites. Still, I feel, that using Elixir is, today, still a large bet. You need to convince your colleagues as much as your bosses / customers to take the risk. But you can rest assured it will not fail you as you need to push it to the next level. Nothing comes for free, and at the right scale, even the Erlang VM is not a silver bullet and will require your engineering team to invest their talent, time and effort to fine tune it. Yet, once you dig deep enough into it, you'll find plenty of ways to solve your problem at a lower cost as compared to other solutions. I see a bright future for Elixir, and a breath of fresh air for Erlang. It's such a great time to be alive!
Kubernetes 在用重量级容器笨拙地模仿 OTP 的轻量进程容错模型，重启成本高得多。
作为一个写了两年 Elixir 的程序员，我觉得很有趣：Kubernetes 社区竭力模仿 Erlang 的 OTP 体系，甚至给 Golang 加更严格的类型系统，试图重新发明'让它崩溃再重启'的理念。但我并不相信'让它崩溃'适用于整个容器——一个容器重启要 20 多秒，而 Erlang/Elixir 的微型进程崩溃后可半自动恢复。想象一个容器内存队列里有 5000 个事件却只处理了 50-100 个就崩溃，或者 100 个进行中的事务被取消。K8s 追求的是容错而非极致速度，容器重启以秒计，Golang 的速度优势又有什么意义呢？
pdimitar查看英文原文
A corollary: As a mostly Elixir programmer for 2 years now, I find it quite amusing how the Kubernetes community tries their damnest to emulate Erlang's OTP system -- going as far as to bolt stricter typing system on Golang even -- and try and reinvent Erlang's "let it crash and get rebooted" idea. I am however not at all convinced that "let it crash" is a good mantra when applied to entire containers. One such container can take 20+ seconds to restart and you can lose a lot more compared to the mini-processes Erlang/Elixir have which are happily left to crash and [semi-]auto-recover. Imagine if that container was ingesting events and crashed when it had 5_000 in its in-memory queue and only managed to process 50-100 of them. Or imagine 100 transactions in progress being cancelled. I applaud the hard work of the Kubernetes team, I am just not very sure they invest their energy in the right tech stack. Sure Golang is faster than Erlang/Elixir -- by a lot, too. But is it not K8s idea to be fault-tolerant and not immensely quick? What does a Golang's speed matter when a container needs seconds to reboot? In my eyes, K8s could have been written in Bash shell scripts... but I am probably missing something important.
历次讨论
同类还值得读
TTY 揭秘（2008）
The TTY demystified (2008)
经典文章深入剖析了 Unix/Linux 中 TTY 子系统的工作原理，包括线路规程、终端驱动、信号与控制字符等机制。
系统与基础设施上首页 11 次 · 2009–2025linusakesson.net编辑精选为什么 GNU grep 这么快
Why GNU grep is Fast
FreeBSD 邮件列表上一篇广为流传的帖子，解释 GNU grep 为什么快：核心在于 Boyer-Moore 字符串搜索算法配合精心优化的内部循环和 I/O，让程序在搜索时“…
系统与基础设施上首页 9 次 · 2010–2022lists.freebsd.orgXplain：为普通人讲解 X11
Xplain – Explaining X11 for the rest of us
《Xplain》是作者（HN 用户 Jasper_）写的一系列 X11 长文，用他在浏览器中重建的 X11 绘图模型做交互演示，讲解 X11 的绘图模型与窗口系统设计。
系统与基础设施上首页 6 次 · 2013–2020magcius.github.io编辑精选旋风教程：为 Linux 创建极小 ELF 可执行文件（1999）
A Whirlwind Tutorial on Creating Really Teensy ELF Executables for Linux (1999)
breadbox 写于 1999 年的经典教程，主题是如何为 Linux 手工构造并裁剪出体积极小的 ELF 可执行文件。
系统与基础设施上首页 6 次 · 2007–2025muppetlabs.comPostgreSQL 内部原理
The internals of PostgreSQL
《The Internals of PostgreSQL》是一本免费在线书籍，系统讲解 PostgreSQL 的内部实现，包括查询处理、并发控制、WAL、缓冲区管理等核心机制。
系统与基础设施上首页 5 次 · 2017–2025interdb.jp编辑精选