系统与基础设施danluu.com
文件操作充满危险
Files Are Fraught with Peril
2019 — 2025，它一次次回到首页
黑点是上首页（≥ 50 分）的那次投稿，空心点是没冲上去的投稿，点越大分数越高。
导读
这篇文章讨论文件和文件系统 API 为什么充满陷阱：写入、同步、重命名等看似基础的操作，即使由数据库、版本控制等专业软件来做也容易出错。文中借用 Dropbox 等案例说明，跨文件系统的文件名编码、元数据、通知机制等差异会让备份/恢复和正确性保证变得极其复杂。作者还质疑 fsync 后 rename 是否真能提供人们以为的原子性。
为什么经久不衰：它把最日常的文件操作拆成一个个隐蔽的失败模式，提醒即使专家也难以正确使用文件系统 API，因此多年后仍值得读。
HN 评论精选
Dropbox 的真正难题是跨文件系统的元数据、编码和语义差异，而非数据损坏；作者用 Dropbox 说明更广泛的文件系统问题。
我做过备份方案并支持过 Dropbox。我认为作者只是拿 Dropbox 当引子，来支撑他对文件系统的担忧，而这些担忧与 Dropbox 的具体案例无关。从文件系统角度看，Dropbox 并不那么担心你因数据损坏而丢数据，因为云端本来就有安全性和版本。它真正代价高昂的问题是元数据：不同文件系统对文件名的命名、编码/字符集支持不同，对扩展元数据或用户权限也有各自的限制；有的支持文件变更通知，有的不支持，最后修改时间的精度也各不相同。Dropbox 的目标是完整备份和恢复系统数据，还要支持从一种文件系统备份、恢复到另一种，组合起来就是噩梦。举个简单例子：文件名里的“ü”按 Unicode 规范化可以存成单个字符，也可以存成“u”加组合分音符；多数用前者，Mac HFS 用后者。
greatgib查看英文原文
I did work on backup solutions and supported Dropbox, and I think that the author used Dropbox as a reference to support his own concerns about filesystems that are not related in any case to the Dropbox case. From the filesystem point of view, I don't think that Dropbox is so much concerned about you losing your data because of corruption. As, anyway they are supposed to be safe and versioned in the cloud. And I think that their app is very simple, ie like no specific 'driver' hack. Their real costly problem is related to metadata! All different file systems support different naming, encoding or character sets for file names. Also, they all have their specificities and limitations regarding the support of extended metadata or user right info. Also some will support file change notification or not, some will have a valid file last modification value that can have different level of decimal precision, etc... So, as Dropbox ambition is that their app can fully backup and restore your system data, supporting backuping from one and restoring to another or the combinatory of all possible for each case is a nightmare. Just a simple example: let suppose that you have a filename with the char "ü". With unicode normalization, this can be stored as a single character or as 2 characters "u" + ".." (letter + particule). Everyone uses the first version, but Mac HFS uses the second form. The crazy thing is that if you try to save a file with the first form in HFS, it will accept but will silently convert the filename. So, let's suppose that it was Dropbox asking to restore such a filename with the letter as a single character, later when listing your files to see if they are in sync, it will see another filename (based on filename bytes) but not the one it expects. So it might want to download it again and again and again if it was not smart.
文件系统 API 设计糟糕到连专家级软件也普遍存在写入 bug，只有 SQLite 某模式例外。
这很吸引人，有点像看事故现场。一个很好的总结是：文件系统 API 设计显然有问题，因为专门使用它们的人都无法正确使用。Pillai 等人在 OSDI’14 上检查了大量写文件的软件，包括数据库和版本控制系统这些我们本以为会安全写文件的软件……他们发现，除 SQLite 在某种特定模式下以外，每一个被测软件都至少有一个 bug。开发 Leveldb、LBDM 等的人比绝大多数程序员更懂文件系统，却仍不能每次都安全地使用文件。
ovi256查看英文原文
This is fascinating, a bit in the same way that looking at accidents is interesting. A good synthesis is: filesystem API design is obviously a problem, given that people that specialize in using them can't do it correctly: "Pillai et al., OSDI’14 looked at a bunch of software that writes to files, including things we'd hope would write to files safely, like datbases and version control systems ... they found that every single piece of software they tested except for SQLite in one particular mode had at least one bug ... programmers who work on things like Leveldb, LBDM, etc., know more about filesystems than the vast majority of programmers ... they still can't use files safely every time"
作者质疑文章关于 fsync+rename 不安全的说法，指出 glibc 文档和 LWN 都暗示该技巧安全，希望找到权威讨论。
哪里能了解更多关于 rename() 技巧并不安全的内容？我也曾（错误地）以为，在 fsync 文件内容后再重命名，是执行“原子”文件写入的有效机制：写入要么完整出现在目标路径，要么产生零字节文件，要么不发生；即使崩溃也不应产生内容不正确的非零字节文件。这个做法在现实中确实被使用，所以它并不正确让我很意外。我甚至找到一篇最近的 LWN 文章暗示相反，看来这个话题有很多混乱。希望能得到更多细节或相关邮件列表链接。补充：glibc 文档也暗示 rename 技巧是安全的……（引用其说明和 Ted Tso 的 LWN 文章）
quickquestion42查看英文原文
Where can I learn more about the rename() trick not being safe? I was also (incorrectly) assuming that renaming a file after having fsynced the contents is a valid mechanism to perform "atomic" file writes where the write either appears completely at the destination path, creates a zero byte file or does not happen at all. But it should never produce an "incorrect" non-zero byte file - even in the face of crashes. It is definitely something that is used in the wild so it was surprising to me that it is not correct. I even found a recent LWN article suggesting otherwise, so there seems to be a lot of confusion on this topic. Would love to learn more if anybody has detailed infos or a link to the relevant mailing list thread or similar. EDIT: Glibc documentation also implies that the "rename trick" is indeed safe: From https://www.gnu.org/software/libc/manual/html_node/Renaming-Files.html > One useful feature of rename is that the meaning of newname changes “atomically” from any previously existing file by that name to its new meaning (i.e., the file that was called oldname). There is no instant at which newname is non-existent “in between” the old meaning and the new meaning. If there is a system crash during the operation, it is possible for both names to still exist; but newname will always be intact if it exists at all. Another relevant source that suggests the trick is safe is https://lwn.net/Articles/327601/. Keep in mind that Ted Tso is the ext4 maintainer. > For the longer term, Ted asked: should the above-described fixes become a part of the filesystem policy for Linux? In other words, should application developers be assured that they'll be able to write a file, rename it on top of another file, omit fsync(), and not encounter zero-length files after a crash? The answer turns out to be "yes," but first Ted presented his other long-term ideas.
询问 ZFS 是否避免非原子重命名，并认为 ZFS 是少有的兼具多平台和数据完整性的可靠选择。
有人知道 ZFS 是否也受重命名非原子性的影响吗？既然 ZFS 已经是我们唯一合理的文件系统候选[+]……如果结论只是“告诉大家用 ZFS”，我会没那么难过。[+] 一个合理文件系统的最低要求是能在多个主流平台工作，并具备校验和等数据完整性措施；这还没算快照、加密等仍然很令人向往的东西。
patrec查看英文原文
Does anyone happen to know if ZFS also suffers from renames being non-atomic? Since ZFS is the only candidate for a reasonable file system we have anyway[+], I'd be a lot less sad if it turns out that way out of this dumpster fire is just tell people to use ZFS. [+] The absolutely minimal requirement for a reasonable file system is that it works on multiple popular platforms and performs checksumming and other data integrity measures. And that's before we even get to still highly desirable stuff like snaphotting, encryption, etc.
历次讨论
同类还值得读
TTY 揭秘（2008）
The TTY demystified (2008)
经典文章深入剖析了 Unix/Linux 中 TTY 子系统的工作原理，包括线路规程、终端驱动、信号与控制字符等机制。
系统与基础设施上首页 11 次 · 2009–2025linusakesson.net编辑精选为什么 GNU grep 这么快
Why GNU grep is Fast
FreeBSD 邮件列表上一篇广为流传的帖子，解释 GNU grep 为什么快：核心在于 Boyer-Moore 字符串搜索算法配合精心优化的内部循环和 I/O，让程序在搜索时“…
系统与基础设施上首页 9 次 · 2010–2022lists.freebsd.orgXplain：为普通人讲解 X11
Xplain – Explaining X11 for the rest of us
《Xplain》是作者（HN 用户 Jasper_）写的一系列 X11 长文，用他在浏览器中重建的 X11 绘图模型做交互演示，讲解 X11 的绘图模型与窗口系统设计。
系统与基础设施上首页 6 次 · 2013–2020magcius.github.io编辑精选旋风教程：为 Linux 创建极小 ELF 可执行文件（1999）
A Whirlwind Tutorial on Creating Really Teensy ELF Executables for Linux (1999)
breadbox 写于 1999 年的经典教程，主题是如何为 Linux 手工构造并裁剪出体积极小的 ELF 可执行文件。
系统与基础设施上首页 6 次 · 2007–2025muppetlabs.comPostgreSQL 内部原理
The internals of PostgreSQL
《The Internals of PostgreSQL》是一本免费在线书籍，系统讲解 PostgreSQL 的内部实现，包括查询处理、并发控制、WAL、缓冲区管理等核心机制。
系统与基础设施上首页 5 次 · 2017–2025interdb.jp编辑精选