系统与基础设施mina.naguib.ca
有时候就是连不上的小小 SSH
The little ssh that (sometimes) couldn't
2012 — 2024，它一次次回到首页
黑点是上首页（≥ 50 分）的那次投稿，空心点是没冲上去的投稿，点越大分数越高。
导读
作者记录了一桩诡异至极的 SSH 故障：连接有时能正常建立、日常小交互也没问题，但一旦有较大的数据发送出去，会话就会卡死。经过层层排查，问题最终追溯到网络层——与 MTU／大包分片相关的链路问题，而不是 SSH 或服务器本身。文章的价值在于展示了一条完整的、逐层排除的调试思路。
为什么经久不衰：它把「网络故障排查」写成了一则侦探故事，示范了如何从表象推断到 MTU 这类底层原因，二十多年后仍是运维与网络工程师的经典参考。
HN 评论精选
MTU 设置过小（1200 字节）遇上 IPv6 不分片规则，导致摇滚乐包更大而被丢弃——生动的 MTU 故障案例。
既然说到这儿，我也讲一个关于 MTU 的离奇经历（虽然没那篇那么离奇）。我曾帮一位用户排查一个在局域网里跑的音频流应用。那位用户能流畅播放古典音乐，却放不了摇滚——真的。古典没问题，放摇滚几分钟后连接就断。这个应用把音频切块、用无损编码压缩，然后每块单独放进一个 UDP 包发到对端。它尽量用 IPv6，因为在局域网里更可靠，必要时也会用 IPv4。跟这位用户来回做了一大堆枯燥的排查后，我终于搞明白了。他不知怎么把网卡的 MTU 设成了 1200 字节。IPv6 在 MTU 低于 1280 字节时不会做 IP 层自动分片，所以更大的包根本发不出去。应用一旦要发超过 1200 字节的音频包就会报错，然后退出连接。为什么只有摇滚出问题？其实很简单：无损编码必然是可变码率，而古典音乐比摇滚压缩得好。放古典时每块音频稳定压到 1200 字节以下，摇滚则偶尔会产生超过阈值的包。这位用户不知道自己的 MTU 为什么被调低，也不需要这么低的 MTU，所以我们把它改回去了……
mikeash查看英文原文
Well, while we're at it, here's my crazy MTU-related war story, although not as crazy as that one! I was troubleshooting with a user of an audio streaming application running over a LAN. The user could stream classical music but not rock music. Seriously. Classical was fine, but when streaming rock, the connection would drop after a few minutes. The application took chunks of audio, compressed them with a lossless codec, and then sent each chunk in a separate UDP packet to the other end. It tried to use IPv6 whenever possible because it was generally more reliable in the LAN environment, although it would happily use IPv4 if need be. After a huge amount of boring troubleshooting going back and forth with this guy, I finally figured it out. Somehow, he had set his network interface's MTU to 1200 bytes. IPv6 won't perform automatic IP-level fragmentation for MTUs below 1280 bytes, so larger packets simply could not be sent at all. The streaming application would try to send an audio packet larger than 1200 bytes, get an error, and bail out of the connection. Why did it only happen with rock music? Turns out to be pretty simple. Lossless codecs are necessarily variable bitrate, and classical music compresses better than rock music. When streaming classical, each chunk of audio consistently compressed to less than 1200 bytes, but rock music produced occasional packets over the threshold. The user didn't know why his MTU was turned down and didn't need it, so we turned it back up and everything worked just fine.
列举三个类似的诡异网络故障（帧中继 MTU、ECN 被防火墙拦、Solaris 自删 ARP），衬托原文之复杂。
这太不可思议了。我职业生涯里最接近的几种情况：1）一个私有帧中继网络某天开始不再传输超过某个尺寸的包。两端都调低 MTU 来绕过，直到我说服帧中继提供商承认问题出在他们网络里。这个诊断相对直接，但那种「能 ssh 登录，一旦执行发全尺寸包的操作（cat 大文件、在大目录里 ls -l）连接就挂住」的感觉还是很怪。2）我写的一个寻呼网关程序（邮件转短信），在 Mac 上测试正常，在 Linux 上却连不上某个 Verizon 网站。原因是 Linux TCP 栈默认开了 ECN，而当时那个 Verizon 网站后面有个有 bug 的防火墙，会丢弃任何带 ECN 位的包。3）一台 Solaris 机器能被连上，但不是每次都能。原因是有人把它自己的 MAC 地址从 ARP 表里删了（Solaris 上确实可以这么干），于是它不再回应针对自己的 ARP 请求。据我回忆，它能主动向外连接，然后你可以从那个对端连回来，直到对端 ARP 条目超时，之后就再也连不上了。这些都没有这篇文章里的情形复杂。
js2查看英文原文
This is insane. The closest scenarios to this I've seen in my career: 1) A private frame relay network that one day stopped passing packets over a certain size. Worked around by lowering the MTU at both ends till I was able to convince the frame relay provider that yes, the problem was in their network. This was relatively straight-forward to diagnose, but it was still odd being able to ssh into a box, then have the connection hang once I did something that sent a full-size packet (cat a large file, ls -l in a big directory, etc). 2) A paging gateway program I wrote (email to SMS) that worked fine when testing on my Mac, but couldn't establish connections to a particular Verizon web site when I ran it from a Linux box. Turned out that the Linux TCP stack had ECN enabled and at the time the Verizon website was behind a buggy firewall that blocked any packets with ECN bits set. 3) A Solaris box that could randomly be connected to, but not always. Turned out someone had deleted its own MAC address from its ARP table (yes, you can do this with Solaris) so it wasn't replying to ARP packets for itself. As I recall, it could make outbound connections, and then you could connect to it from that same peer until the peer timed out the ARP entry. Then the peer couldn't reach the Solaris box again. None of these are nearly as complex as the scenario in this story.
T1 某段误配 AMI 而非 B8ZS，导致 JPG 这类缺乏 0/1 跳变的数据失去时钟同步而损坏。
让我想起当年一条 T1 线路损坏数据包的问题。在远端启用第二条 T1 后不久，我们发现远端有些网页会显示坏掉的 JPG 图片。排查后发现只有走新 T1 的流量才这样，旧 T1 一切正常。ping 和其他 IP 流量走哪条线都正常，但走新 T1 的图片就是显示不出来。我们好几次向电信确认 T1 配置正确、我方设备参数匹配，可新 T1 上仍有错位的数据位。最后电信检查了这条长距离 T1 电路的每一段参数，终于发现有一段被配置成了 AMI 而不是 B8ZS（如果我没记错的话，总之是某段配置错了）。结果就是某些不包含足够 0/1 跳变的用户数据模式会导致该段失去时钟同步、数据包被破坏——而这种模式最常出现在 JPG 里。他们改好那段的参数后，一切恢复正常。那次真是挠破了头，而且很挫败：底层电信的人根本不相信我们这些搞二层三层的互联网人能准确诊断出他们一层网络的问题。
gwright查看英文原文
Reminds me of a problem I had with a T1 circuit corrupting packets. Shortly after bringing up a second T1 into a remote location we discovered that some web pages would show broken JPG images at the remote site. Some troubleshooting revealed that this only happened when traffic was routed over the new T1. The old T1 worked just fine. Pings, and other IP traffic seemed to work over either line but we kept seeing the broken image icon for some reason when traffic came over the new T1. We tried several times to confirm with the telco that the T1 was provisioned correctly and that our equipment matched those telco parameters. Still had some mangled bits going over that new T1. Finally had the telco check the parameters over every span in the new (long-distance) T1 circuit and they eventually found one segment that was configured for AMI instead of B8ZS (if I can remember correctly, certainly it was a misconfigured segment though). The net result is that certain user-data patterns that didn't include sufficient 0/1 transitions would lead to loss of clock synchronization over that segment and corrupted packets. Those patterns were most likely to occur in JPGs. Once they corrected the parameters on that segment, everything worked as expected. Quite a bit of head scratching with that one and lots of frustration as the layer-1 telco culture just couldn't comprehend that layer-2/3 Internet folks could accurately diagnose problems with their layer-1 network.
强调文中调试决策树对运维的价值，并分享 Blekko 交换机吞掉特定位模式 UDP 分片的类似经历。
这个故事太棒了。如果你做 devops，我建议你研究一下这个事件序列，尤其是那棵调试决策树。你不总能拿到所有机器的访问权限，但你可以通过逐台机器穿过去来构造「视图」，就像天文学家利用引力透镜那样。我们在 Blekko 也遇到过类似问题：用的一台 10G 交换机不肯放过 UDP 分片里某种特定的位模式，包就那么消失了。烦人得很，最后的解决办法是在重试时往包里加随机数据，好让至少有一个数据报能完整通过。
ChuckMcM查看英文原文
That is an awesome story. If you're in devops I would suggest you look at the sequence of events, especially the debugging decision tree. You can't always get access to all of the machines but you can create 'views' by going through them. Sort of like astronomers using a gravitational lens. We had a similar issue at Blekko where a 10G switch we were using would not pass a certain bit pattern in a UDP packet fragment. Just vanished. Annoying as heck, the fix was to add random data to the packet on retries so that at least one datagram made it through intact.
历次讨论
同类还值得读
TTY 揭秘（2008）
The TTY demystified (2008)
经典文章深入剖析了 Unix/Linux 中 TTY 子系统的工作原理，包括线路规程、终端驱动、信号与控制字符等机制。
系统与基础设施上首页 11 次 · 2009–2025linusakesson.net编辑精选为什么 GNU grep 这么快
Why GNU grep is Fast
FreeBSD 邮件列表上一篇广为流传的帖子，解释 GNU grep 为什么快：核心在于 Boyer-Moore 字符串搜索算法配合精心优化的内部循环和 I/O，让程序在搜索时“…
系统与基础设施上首页 9 次 · 2010–2022lists.freebsd.orgXplain：为普通人讲解 X11
Xplain – Explaining X11 for the rest of us
《Xplain》是作者（HN 用户 Jasper_）写的一系列 X11 长文，用他在浏览器中重建的 X11 绘图模型做交互演示，讲解 X11 的绘图模型与窗口系统设计。
系统与基础设施上首页 6 次 · 2013–2020magcius.github.io编辑精选旋风教程：为 Linux 创建极小 ELF 可执行文件（1999）
A Whirlwind Tutorial on Creating Really Teensy ELF Executables for Linux (1999)
breadbox 写于 1999 年的经典教程，主题是如何为 Linux 手工构造并裁剪出体积极小的 ELF 可执行文件。
系统与基础设施上首页 6 次 · 2007–2025muppetlabs.comPostgreSQL 内部原理
The internals of PostgreSQL
《The Internals of PostgreSQL》是一本免费在线书籍，系统讲解 PostgreSQL 的内部实现，包括查询处理、并发控制、WAL、缓冲区管理等核心机制。
系统与基础设施上首页 5 次 · 2017–2025interdb.jp编辑精选