编程与软件工程apenwarr.ca
mtime 比较被认为有害
Mtime comparison considered harmful
2018 — 2022,它一次次回到首页
黑点是上首页(≥ 50 分)的那次投稿,空心点是没冲上去的投稿,点越大分数越高。
导读
作者 apenwarr 深入剖析了用文件 mtime(修改时间)判断文件是否需要重新构建的种种陷阱:时钟跳变、精度不足、硬链接、目录 mtime 不随文件内容更新等问题,论证 mtime 比较机制“有害”。文章还讨论了校验和方案的缺陷,并介绍了其 redo 构建系统采用的 mtime 加上其他信号混合判断的思路。
为什么经久不衰:构建系统的增量重建判断是每个开发者都会踩的坑,这篇多年反复登上首页的文章把 mtime 的失效场景讲得最透彻,至今仍是理解构建工具行为的必读参考。
HN 评论精选
构建系统应关注“输出是否反映全部输入状态”,把 mtime 当作弱内容签名来用可绕开时钟问题
在 Ninja 里我也磕磕绊绊遇到过文中同样的问题。我最终意识到,真正关键的问题是“这个输出文件是否反映了所有输入的状态”,而不是 mtime 本身——这里的“输入”不仅包括输入文件内容,还包括生成输出所用的可执行文件和命令行。如果把 mtime/inode 看成输入的一种弱内容签名,那么规则就变成“mtime 和上次不一样就重建”,完全不关心它们的相对大小,从而绕开了时钟偏移的很多问题。一个不错的技巧是把所有输入(文件、命令行等)合并成一个校验和,缺点是构建系统决定重建时很难告诉用户原因,只能说“某处某个输入变了”。
evmar查看英文原文
In Ninja I sorta stumbled through some of the same issues described here. I eventually realized that the interesting question is "does this output file reflect the state of all the inputs" and not anything in particular about mtimes, and that"inputs" includes not only the contents of the input files, but also the executables and command lines used to produce the output. If you squint, mtime/inode etc. behave like a weak content signature of the input. And once you have that perspective, you say "if mtime != mtime I had last time, rebuild", without caring about their relative values, and that sidesteps a lot of clock skew related issues. It does "the wrong thing" if someone intentionally pushes timestamps to a point in the past (e.g. when switching branches to an older branch) as an attempt to game such a system, but playing games with mtime is not the right approach for such a thing, totally hermetic builds are. One nice trick is that you can even capture all the "inputs" with a single checksum that combines all the files/command lines/etc., and that easily transitions between truly looking at file content or just file metadata. The one downside is that when the build system decides to rebuild something, it's hard to tell the user why -- you end up just saying "some input changed somewhere".
作者关于文件系统时间精度和时钟跳变的说法存在事实错误
“流行误解”一节里似乎有作者自己的误解。关于精度,他说“几乎没有文件系统提供纳秒级精度”,但事实恰恰相反:ext4、xfs、btrfs、ZFS 等常见文件系统都支持。他 ext4 系统只有 10ms 粒度,多半是从 ext2/ext3 升级来的,并非默认。另外 NTFS 的粒度是 100ns。至于“系统时钟跳变”一段,如果指 NTP 那是对的;如果指夏令时或时区,那大家内部都用 UTC,不会有影响。
chungy查看英文原文
The "popular misconceptions" section seems to have a couple of the author's own misconceptions. * On precision, he notes "almost no filesystems provide that kind of precision" (nanoseconds), but I would honestly say the exact opposite statement. ext4, xfs, btrfs, ZFS are some of the very common file systems that support this. He cites that his ext4 system only has 10ms granularity, which is most certainly not the default, but likely a result of upgrading from ext2/ext3 to ext4. As an aside, NTFS only has a granularity of 100ns. * It is unclear what he means by "If your system clock jumps from one time to another...". If this is talking about NTP, it's probably accurate. My first reading was "daylight saving" or time zone changes, in which case, everyone uses UTC internally and such changes don't affect the actual mtimes. (You might get strange cases where a file listing regards a file modified at 01:45 to be older than a file modified at 01:20, but if you display in UTC, you can see it's just DST nonsense)
纠正文中关于 Git 索引文件用 mmap 写入的说法,实际是临时文件加原子重命名
文中暗示 .git/index 文件通过 mmap 写入,但这不属实。Git 是把它完整写入一个新的临时文件/锁文件,然后原子重命名到位。Git 从不以 PROT_WRITE 之外的方式使用 mmap——实际上只用 PROT_READ,因为并非所有支持的平台都能写 mmap(兼容回退方案是用 pread() 读进堆缓冲区)。
peff查看英文原文
> the .git/index file, which uses mmap, is synced incorrectly by file sync tools relying on mtime This part implies that the index file is written via mmap, but that's not true. It is fully rewritten to a new tempfile/lockfile, and then atomically renamed into place. Git does not ever mmap with anything but PROT_READ, because not all supported platforms can do writes (in particular, the compat fallback just pread()s into a heap buffer).
目录 mtime 不随文件更新主要是出于性能考虑,而非硬链接
文件内容更新时其所在目录的 mtime 不变,可能部分是因为硬链接,但我怀疑主要是效率问题:如果每次改文件都递归更新到根目录的所有目录,深层目录树下开销会很大,对某些介质还会缩短硬件寿命。我记得即使在很多文件系统默认更新 atime 的年代,也只有当前对象的时间被更新,父目录链不会。
dspillett查看英文原文
> when you update the content of a file, the mtime of its containing directory is not changed ... if the directory mtime were updated (recursively to the root) when contained files changed ... This is probably because of hardlinks It might be partly due to hardlinks, but I suspect it is mostly an efficiency issue. Modifying a file could mean updating quite a few directories every time if you have a deep tree. For some uses this could be quite a performance hit, and for some media would potentially reduce the life of your hardware too. IIRC even when updating atime for every access was the usual default for many filesystems, only the current object had its time updated not all its chain of parents.
历次讨论
同类还值得读
开源应用架构
The Architecture of Open Source Applications
《开源软件架构》(AOSA)是一本在线免费书,邀请知名开源项目的作者亲自剖析其内部设计,涵盖 SQLAlchemy、LLVM、Audacity、Apache、Bash 等系统。
编程与软件工程上首页 9 次 · 2011–2024aosabook.org编辑精选编程语言简史:简短、不完整且多半错误
A Brief, Incomplete, and Mostly Wrong History of Programming Languages
一篇以戏谑、故意不准确的方式回顾编程语言史的博客文章,用夸张轶事和讽刺定义串起 Lisp、C、C++、Java、JavaScript、PHP 等语言。
编程与软件工程上首页 8 次 · 2009–2021james-iry.blogspot.comSQLite 是如何测试的
How SQLite is tested
SQLite 官方文档页,系统介绍 SQLite 的测试体系:多种测试方法(TCL 测试、模糊测试、回归测试、静态分析等)和海量测试用例,展示了这个小型开源项目如何靠极致测试获得…
编程与软件工程上首页 8 次 · 2009–2025sqlite.org编辑精选高级编程语言(2009)
Advanced programming languages (2009)
Matt Might 的经典文章,按用途推荐“高级编程语言”:如果追求就业就选主流语言,如果追求表达力、学术研究或创业,则推荐 Haskell、OCaml、Scala 等函数式语…
编程与软件工程上首页 8 次 · 2009–2022matt.might.net我的 Forth 与栈机器经历
Yosefk - My history with Forth & stack machines
作者以亲身经历回顾了使用 Forth 语言和栈式机器的历程:既赞叹其简洁优雅、代码量小、可自举的优点,也坦率指出栈操作在实际编程中的痛苦,以及 Forth 社区文化中的问题。
编程与软件工程上首页 8 次 · 2010–2024yosefk.com编辑精选