编程与软件工程kunststube.net编辑精选
每位程序员都绝对必须了解的编码知识（2011）
What Every Programmer Absolutely, Positively Needs To Know About Encodings (2011)
2015 — 2022，它一次次回到首页
黑点是上首页（≥ 50 分）的那次投稿，空心点是没冲上去的投稿，点越大分数越高。
导读
这篇文章面向程序员系统讲解字符编码：从字节、码点和字符的区别入手，说明 UTF-8 等编码如何工作，以及处理文本时容易踩的坑。它用具体例子拆解常见误解，帮助读者建立处理乱码和编码转换问题的基本框架。
为什么经久不衰：编码问题至今仍无处不在，而这篇把基础概念和常见陷阱讲透的文章，多年后仍是可靠的心智模型。
HN 评论精选
码点不等于用户可见字符，组合字符会让排序等问题更复杂。
文章一带而过但非常重要的一点是：Unicode 中的一个码点不一定对应屏幕上的一个字符。一个“字符”常常由多个码点组合而成。这又让本已复杂的排序问题更麻烦（谁知道“按字母顺序排列这些名字”居然是个有歧义的说法）。
bruce511查看英文原文
A detail skimmed over in the article, but one which has significant importance is that; "one code point in unicode does not necessarily map to one character on the screen." A "character" can, and often does, get constructed from multiple code points. This doesn't help an already complicated sorting issue (who knew "sort these names alphabetically" could be an ambiguous statement).
显式声明编码，避免浏览器自动猜测导致偶发乱码。
文章应该提到：所有使用 UTF-8 的 HTML 文档都要在开头 1024 字节内放 <meta charset="utf-8" />。否则浏览器会启动“编码猜测器”，有时会猜错。我曾见纽约时报首页被错误转换，因为开头附近唯一的 UTF-8 序列来自 apple-touch-icon 的 144×144 中的乘号 ×，而不是字母 x，把猜测器搞糊涂了。
Animats查看英文原文
Something they should have mentioned: put a <meta charset="utf-8" /> in all your HTML documents that are in UTF-8. Note that this has to be in the first 1024 bytes of the document. Otherwise, the browser has to invoke the "encoding guesser"[1], which will sometimes guess wrong. (W3C: "The user agent may attempt to autodetect the character encoding from applying frequency analysis or other algorithms to the data stream.") The result will be occasional users seeing random pages in the wrong encoding, depending on browser, browser version, platform, and page content. I recently saw the front page of the New York Times misconverted because they didn't specify an encoding, and the only UTF-8 sequence near the beginning of the document was in <link rel="apple-touch-icon-precomposed" sizes="144×144" ... The "×" there is not the letter x, it's the Unicode multiplication symbol. This confused an encoding guesser. Don't go there. [1] http://www.w3.org/TR/html5/syntax.html#determining-the-character-encoding
全链路统一 UTF-8，能显著减少乱码和修复垃圾数据的成本。
我们处理大量多语言文本，对编码问题的简单准则是“用 UTF-8，否则完蛋”。具体来说：应用内所有人类文本只存 UTF-8；其他编码的数据通过包装层转换；持久层只存 UTF-8；公开 API 只接受和输出 UTF-8 并写明编码。
PeterisP查看英文原文
We work with a lot of multilingual text, and for "what to know about encodings and character sets" we have a very simple answer to that - a guideline called "use UTF8 or die". It's not suitable for absolutely everyone (e.g. if you have a lot of Asian text then you may want a different encoding), but for our use case every single deviation causes lot of headache, risks and unneccessary work in fixing garbage data. In simplistic terms what we mean by this guideline: * in your app, 100% all human text should be stored UTF8 only, no exceptions. If you need to deal with data in other encodings - other databases, old documents, whatever - make a wrapper/interface that takes some resource ID and returns the data with everything properly converted to UTF8; and has no way (or at least no convenient way) to access the original bytes in another encoding. * in all persistence layers, store text only as UTF8. If at all possible, don't even provide options to export files in other encodings. If legacy/foreign data stores need another encoding, then in your code never have an API that requires a programmer to pass data in that encoding - the functions "store_document_to_the_mainframe_in_EBCDIC" and "send_cardholder_data_to_that_weird_CC_gateway" should take UTF8 strings only and handle the encodings themselves. * in all [semi-]public API, treat text as UTF8-only and document that. If your API documentation mentions a text field, state the encoding so that there is no guessing or assuming by anyone. * in all system configuration, set UTF8 as the default whenever possible. A database server? Make sure that any new databases/tables/text fields will have UTF8 set as the default, so unless someone takes explicit action then user-local-language encodings won't accidentally appear. * Whoever introduces a single byte of different encoding data is responsible for fixing the resulting mess. This is the key part. Did you write a data input function that passed on data in the user computer default encoding; tested it only on US-ASCII nonenglish symbols; and got a bunch of garbage data stored? You're responsible for finding the relevant entries and fixing them, not only your code. Used a third party library that crashes or loses data when passed non-english unicode symbols? Either fix the library (if it's OS) or rewrite code to use a different one.
Unicode 不仅是码表，还包含文本处理和规范化算法。
文章说“Unicode 只是又一种编码方案”，但它不止是编码：还包含算法，例如 RTL 语言的排序与字形整形、RTL 与 LTR 混排时如何处理（“123.”末尾的点是小数点还是句号，决定它排向左边还是右边），以及如何判断规范化前后的数据是否等价。
peapicker查看英文原文
From the article: "Overall, Unicode is yet another encoding scheme." It is more than that - for instance, it includes algorithms as well... for instance, dealing with RTL languages with ordering and shaping rules (i.e. Arabic), how to know what to do when RTL languages are mixed with LTR (is that '.' at the end of '123' a decimal point, or a period? (determines if it goes to the right or the left or the sequence)) and how to know when data is equivalent despite being normalized or not, etc...
历次讨论
同类还值得读
开源应用架构
The Architecture of Open Source Applications
《开源软件架构》（AOSA）是一本在线免费书，邀请知名开源项目的作者亲自剖析其内部设计，涵盖 SQLAlchemy、LLVM、Audacity、Apache、Bash 等系统。
编程与软件工程上首页 9 次 · 2011–2024aosabook.org编辑精选编程语言简史：简短、不完整且多半错误
A Brief, Incomplete, and Mostly Wrong History of Programming Languages
一篇以戏谑、故意不准确的方式回顾编程语言史的博客文章，用夸张轶事和讽刺定义串起 Lisp、C、C++、Java、JavaScript、PHP 等语言。
编程与软件工程上首页 8 次 · 2009–2021james-iry.blogspot.comSQLite 是如何测试的
How SQLite is tested
SQLite 官方文档页，系统介绍 SQLite 的测试体系：多种测试方法（TCL 测试、模糊测试、回归测试、静态分析等）和海量测试用例，展示了这个小型开源项目如何靠极致测试获得…
编程与软件工程上首页 8 次 · 2009–2025sqlite.org编辑精选高级编程语言（2009）
Advanced programming languages (2009)
Matt Might 的经典文章，按用途推荐“高级编程语言”：如果追求就业就选主流语言，如果追求表达力、学术研究或创业，则推荐 Haskell、OCaml、Scala 等函数式语…
编程与软件工程上首页 8 次 · 2009–2022matt.might.net我的 Forth 与栈机器经历
Yosefk - My history with Forth & stack machines
作者以亲身经历回顾了使用 Forth 语言和栈式机器的历程：既赞叹其简洁优雅、代码量小、可自举的优点，也坦率指出栈操作在实际编程中的痛苦，以及 Forth 社区文化中的问题。
编程与软件工程上首页 8 次 · 2010–2024yosefk.com编辑精选