编程与软件工程hsivonen.fi
Unicode 中的字符串长度
String Lengths in Unicode
2019 — 2025，它一次次回到首页
黑点是上首页（≥ 50 分）的那次投稿，空心点是没冲上去的投稿，点越大分数越高。
导读
这是 Mozilla 工程师 Henri Sivonen 写的一篇长文，梳理“字符串长度”在 Unicode 下到底指什么：字节数、码点（Unicode scalar value）数、还是用户感知的字符（grapheme cluster）数，并对比多种语言/API 的度量方式与各自的坑。文章的核心观点是“长度”本身是个含义模糊的度量，选错口径会在配额限制、内存分配、界面排版等场景引入 bug。
为什么经久不衰：Unicode 的复杂性和“长度”一词的歧义不会随时间消失，任何要处理文本的开发者今天仍会踩同一个坑，而这篇把各家实现摊开对比的文章依然是最好的地图之一。
HN 评论精选
这篇里最接近洞见的其实是引用别人的一句话：“字符串长度大约和字符串高度一样，是个没什么意义的度量。”围绕这句话文章给了三条理由。1）“为了分配内存”：底层代码确实需要把字符串放进内存，这和整数有大小是一回事，但我不会把这个尺寸叫“length”；而且如果 length(5) 在语言里是错误，按这个定义 length("word") 也该是错误。2）“配额上限”：随便挑一个数就行，没理由把你用来做任意配额的值叫“长度”；如果配额有理由，你就该度量那个理由，比如磁盘上的字节数。3）“显示空间”：这个确实是长度，但单位是像素（或毫米等距离单位），而且需要大量额外参数才能算出来——注意这时字符串高度也真的存在，同样需要许多参数。把关心字符串“长度”的代码当作代码异味；如果它出现在底层基础设施（内存分配器或文本渲染器）里，可能像其他底层异味一样有正当理由，但要确保有认真的单元测试并清楚记录为什么需要它。如果它出现在你“很棒”的社交媒体网站后端，那几乎肯定……
tialaramex查看英文原文
The closest to insight in this post is a quote from somebody else. “String length is about as meaningful a measurement as string height” Under that quote three rationale are offered. 1. "to allocate memory". Fine, like an integer your very low level code needs to actually store strings in RAM and in this sense both integers and strings have a size. I don't think I'd name this size "length" though and certainly if length(5) is an error in your language it seems as though length("word") is also an error by this definition. 2. "quota limit". Just pick anything, there is definitely no reason to name the value you're using for this arbitrary quota "length". If you have a _reason_ for the quota, then you need to measure the reason, e.g. bytes on disk. 3. "display space". This one is a length! But it's in pixels (or milimeters, or some other type of distance unit) and it requires a LOT of additional parameters to calculate. And you'll notice that at last there is actually a string height too for this case, which likewise requires many additional parameters to calculate. Treat code that cares about string "length" as code smell. If it's in your low-level fundamentals (whether in the memory allocator or the text renderer) it might, like a lot of other things which smell down there, have a good reason to be there, make sure it's seriously unit tested and clearly documented as to why it's necessary. If it's in your "brilliant" new social media web site backend, it's almost certainly wrong and figuring out what you ought to be doing instead of asking the "length" of a string will probably fix a bug you may not know you have.
Rust 用不同 API 暴露不同长度口径，但即使 Unicode 官方的字素分段规则也需要针对阿拉伯语、波斯语等文本自行调整。
我觉得 Rust 逼你意识到这件事的复杂性是好事。你写 s.len()，得到的是 UTF-8 字节数，可能正是你要的，也可能差得很远；换成 s.chars().count() 得到的是 Unicode 标量值数，可能更接近；而如果你需要按 Unicode 附件 29 定义的“用户感知字符”做真正分段，就得引入外部 crate，这也很合理。但要知道，即便是附件 29，在某种程度上也只是起点：它的分段规则对我处理的波斯语和阿拉伯语文本并不开箱即用。我不完全认同它对零宽不连字（U+200C）的处理，而且它根本没涉及手动的 tatweel/kashida（U+0640）。所以你得按自己的用例做调整，这个兔子洞很深。
theobeers查看英文原文
I think it’s good that Rust sort of forces you to recognize the complexity in this. If you try s.len(), it’ll be in terms of UTF-8 bytes, which might be what you want… or might be far from it. If you switch to s.chars().count(), you’ll get Unicode scalar values, which may be closer to the mark. And if you need proper segmentation by “user-perceived character,” as set out in Unicode Annex 29, you’ll have to bring in an external crate. That’s fair enough. Keep in mind that even Annex 29 is, on some level, just a starting point. Its segmentation rules don’t work “out of the box” for me with Persian and Arabic text. I’m not totally on-board with its treatment of the zero-width non-joiner (U+200C), and it doesn’t deal with manual tatweel/kashida (U+0640) at all. So you make the necessary adjustments for your use case. The rabbit hole is deep.
Unicode 输入输出不难，难在测量渲染尺寸和编码转换，作者佩服之余也希望得到工程实践上的指引。
我真心佩服作者如此细致地钻研 Unicode 并比较不同语言的多种实现，这肯定花了不少时间！他在 Mozilla 工作，估计确实需要了解这些细节，我自己实在没耐心去分析。某种程度上我很怕 Unicode：编程语言能让文本框和配置文件输入输出 Unicode 我不在乎，难的是当你真的需要知道字符串渲染后的尺寸，或者输出格式要求做编码转换的时候。出问题的地方太多了，文中只有一小段提到字体，而字体编码、字距调整之类的“恶龙”还很多。想象你在做游戏，ASCII 一切顺畅，然后老板要你把游戏发行到亚洲市场——天哪，这就是我的噩梦！该怎么办？选一门什么都帮你搞定的语言（哪门？）、用专门的库？字体渲染又怎么办？有什么推荐吗？
zubspace查看英文原文
I seriously applaud the writer to dive into Unicode in such detail and compare multiple implementation in different languages. That must have taken a while! He is working for Mozilla and I guess he needs to actually know all those nitty-gritty details. I could not imagine myself to even bring up the patience for analyzing it. In some way I am really scared about Unicode. I don't care if the programming language simply allows input and output of Unicode in text-fields and configuration files. But where it gets tough is, if you actually need to know the rendered size of a string or convert encodings, if the output format requires it. There are so many places where stuff can go awry and there's only a small passage in the blog post which mentions fonts. There are multiple dragons abound, like font encoding, kernings and what not. Imagine writing a game. Everything works nice and dandy with your ASCII format and now your boss approaches you and wants to distribute the game for the asian market! Oh, dear... My nightmares are made out of this! How do you handle this? Do you use a programming language which does everything you need? (Which one?) Do you use some special libraries? What about font rendering? Any recommendations?
把 emoji 放进单元测试，是防止西文开发者对输入做错误假设的实用手段。
以英语为母语、主要面向西文做应用的开发者，很容易引入编码方面的国际化 bug，这对其他语言的用户很不公平。Emoji 在日常文本中的流行其实是件好事，它逼着开发者去处理 Unicode 光谱的高端，少对输入做想当然的假设。在数据处理类应用里，我经常会在单元测试里塞几个 emoji。
cmancini查看英文原文
English-native developers building apps for mainly Western languages can easily introduce encoding i18n bugs that are really unfair for other folks. The rise of emoji in everyday text has been great to force developers to deal with the upper end of the unicode spectrum and make fewer assumptions about inputs. Often in a data processing app I'll throw a few emoji in my unit tests.
历次讨论
同类还值得读
开源应用架构
The Architecture of Open Source Applications
《开源软件架构》（AOSA）是一本在线免费书，邀请知名开源项目的作者亲自剖析其内部设计，涵盖 SQLAlchemy、LLVM、Audacity、Apache、Bash 等系统。
编程与软件工程上首页 9 次 · 2011–2024aosabook.org编辑精选编程语言简史：简短、不完整且多半错误
A Brief, Incomplete, and Mostly Wrong History of Programming Languages
一篇以戏谑、故意不准确的方式回顾编程语言史的博客文章，用夸张轶事和讽刺定义串起 Lisp、C、C++、Java、JavaScript、PHP 等语言。
编程与软件工程上首页 8 次 · 2009–2021james-iry.blogspot.comSQLite 是如何测试的
How SQLite is tested
SQLite 官方文档页，系统介绍 SQLite 的测试体系：多种测试方法（TCL 测试、模糊测试、回归测试、静态分析等）和海量测试用例，展示了这个小型开源项目如何靠极致测试获得…
编程与软件工程上首页 8 次 · 2009–2025sqlite.org编辑精选高级编程语言（2009）
Advanced programming languages (2009)
Matt Might 的经典文章，按用途推荐“高级编程语言”：如果追求就业就选主流语言，如果追求表达力、学术研究或创业，则推荐 Haskell、OCaml、Scala 等函数式语…
编程与软件工程上首页 8 次 · 2009–2022matt.might.net我的 Forth 与栈机器经历
Yosefk - My history with Forth & stack machines
作者以亲身经历回顾了使用 Forth 语言和栈式机器的历程：既赞叹其简洁优雅、代码量小、可自举的优点，也坦率指出栈操作在实际编程中的痛苦，以及 Forth 社区文化中的问题。
编程与软件工程上首页 8 次 · 2010–2024yosefk.com编辑精选