编程与软件工程prog21.dadgum.com
拼写检查器曾是软件工程的重大壮举(2008)
A spellchecker used to be a major feat of software engineering (2008)
2012 — 2025,它一次次回到首页
黑点是上首页(≥ 50 分)的那次投稿,空心点是没冲上去的投稿,点越大分数越高。
导读
这篇 2008 年的文章回顾了拼写检查器曾经是软件工程中的一大难题:在内存只有几十 KB 的年代,如何把数万词的词典塞进软盘并快速查找,需要大量巧妙的压缩和算法设计。文章展示了早期工程师在极端资源限制下的创造力。
为什么经久不衰:它提醒我们许多今天被视为“已解决”的问题背后都有精巧的工程史,在资源受限的嵌入式开发中这些思路依然有借鉴价值。
HN 评论精选
补充介绍了一种不依赖词典、基于字符三元组统计的拼写检查方法
这里有一篇更详细的文章(链接)。我特别对其中一种不用词典的方法感兴趣:它把文本切分成三元组,统计每个词的“怪异指数”——比如 pkxie 几乎不含任何常见三元组,会被标为高度怪异;而 fairy 的 fai、air、iry 都很常见,指数就低。它虽然抓不住普通拼写错误,但很擅长发现打字错误,而且不限于英语,法语、德语、希腊语都适用。(文中的 Morris 就是 Bob Morris。)
DanBC查看英文原文
Here's an article that people might be interested in. It gives a bit more detail: https://web.archive.org/web/20100706052342/http://www.spellingsociety.org/journals/j20/spellchecking.php I'm particularly interested in this one, and I'm curious about how useful something like this would be to use. > The second does not use a dictionary at all (Morris & Cherry 1975). Like the previous method, it divides the text into trigrams, but it creates a table of these, noting how often each one occurs in this particular piece of text. It then goes through the text again calculating an index of peculiarity for each word on the basis of the trigrams it contains. Given pkxie, for instance, it would probably find that this was the only word in the text containing pkx and kxi (and possibly xie too), so it would rate it highly peculiar. The word fairy, by contrast, would get a low rating since fai, air and iry probably all occur elsewhere, perhaps quite often, in the passage being analysed. Having completed its analysis, it draws the user's attention to any words with a high peculiarity index. Like the previous method, it would fail to spot a high proportion of ordinary spelling errors, but it is quite good at spotting typing errors, which it was designed for. An advantage that it has over all dictionary-based methods is that it is not tied to English; it will work on passages of, say, French, German or Greek. (The Morris there is Bob Morris).
亲历者分享当年如何利用词间前缀相似性把 7 万词词典压缩到每词不到 2 字节
我在 1980 年代写过拼写检查器。当时在利兹的 Tasman 公司,用 8086 汇编为 IBM PC 兼容机开发文字处理软件,之后做了拼写检查器。我对 Collins 的 7 万词表做了大量分析,生成 token 来表示常见字母串,但为了塞进一张 360K 软盘还是不得不大幅删减词表。离开公司后某天夜里躺在床上,我意识到自己哪里错了:7 万个词里除了 26 个,几乎每个词都和前一个词共享大部分字符。于是用第一个字节的前 5 位表示与前词相同的字符数,3 位表示常见结尾(s、ing 等),其余字节存 token,最终把词表压缩到每词不到 2 字节。后来 Tasman 把全部 7 万词和检查器装进了 ZX Spectrum +3 的一张 175K 软盘。
RegW查看英文原文
I wrote a spelling checker in the 1980's In my first job I worked for Tasman in Leeds and produced a Word Processor for IBM PC compatibles in 8086 assembler with some help, and then a spelling checker. For the spelling checker I did a whole load of analysis on a 70,000 word list from Collins and produced a list of tokens to represent common strings of letters. However, in the end I really had to cut the original word list down to get the whole thing onto a single 360K floppy. After I left Tasman, I was lying in bed one night still thinking about it and realised where I had gone wrong. The tokenising thing, which someone else had put me onto had blinded me. I had stared at word lists for months and hadn't pinned down the obvious pattern. All but 26 words in the 70,000 word list shares the bulk of their characters with the word before it. So the solution is use 5 bits of the first byte as a count of chars from the word before, 3 bits indicate commons enddings (ship, s, ing) or that the following bytes are tokens for the rest of the word. With this I got the word list compressed to less than 2 bytes per word. I took this back to Tasman. They put all 70,000 words and the spelling checker onto a 175K floppy for the ZX Spectrum +3. [Edited for typos]
拼写检查至今仍是难题,只是被成熟库掩盖了
尽管硬件进步巨大,一个好的拼写检查器至今仍是困难的工程问题。光靠一个哈希表是行不通的。人们之所以觉得它已被解决,只是因为有现成的库可用。比如开源的 Hunspell 就远超一万行代码。我会说四种语言,就我的经验而言,微软 Office 的拼写检查是我用过最好的。
Const-me查看英文原文
A good spell checker is still a hard engineering problem, despite the hardware progress. Just a hash map ain't gonna work. The only reason spell checking is perceived as a solved problem is availability of libraries. Here's an open source example https://github.com/hunspell/hunspell way above 10k lines of code. I speak 4 languages, and in my experience what's in Microsoft Office is the best one I used so far.
大规模在线拼写纠错(如谷歌搜索)依然是艰巨的工程挑战
谷歌的拼写检查器至今仍是一项重大的软件工程壮举 ;-)它必须处理所有语言,不断应对新造的词,还要有足够低的延迟和足够高的吞吐量来处理每一次网页搜索(当然,缓存大概解决了三分之二的流量)。
lacker查看英文原文
Google's spell checker still is a major feat of software engineering ;-) It must handle every language, handle new words constantly being coined, have low enough latency and high enough throughput to run it on every web search. (Well, caching probably solves 2/3 of traffic.)
历次讨论
同类还值得读
开源应用架构
The Architecture of Open Source Applications
《开源软件架构》(AOSA)是一本在线免费书,邀请知名开源项目的作者亲自剖析其内部设计,涵盖 SQLAlchemy、LLVM、Audacity、Apache、Bash 等系统。
编程与软件工程上首页 9 次 · 2011–2024aosabook.org编辑精选编程语言简史:简短、不完整且多半错误
A Brief, Incomplete, and Mostly Wrong History of Programming Languages
一篇以戏谑、故意不准确的方式回顾编程语言史的博客文章,用夸张轶事和讽刺定义串起 Lisp、C、C++、Java、JavaScript、PHP 等语言。
编程与软件工程上首页 8 次 · 2009–2021james-iry.blogspot.comSQLite 是如何测试的
How SQLite is tested
SQLite 官方文档页,系统介绍 SQLite 的测试体系:多种测试方法(TCL 测试、模糊测试、回归测试、静态分析等)和海量测试用例,展示了这个小型开源项目如何靠极致测试获得…
编程与软件工程上首页 8 次 · 2009–2025sqlite.org编辑精选高级编程语言(2009)
Advanced programming languages (2009)
Matt Might 的经典文章,按用途推荐“高级编程语言”:如果追求就业就选主流语言,如果追求表达力、学术研究或创业,则推荐 Haskell、OCaml、Scala 等函数式语…
编程与软件工程上首页 8 次 · 2009–2022matt.might.net我的 Forth 与栈机器经历
Yosefk - My history with Forth & stack machines
作者以亲身经历回顾了使用 Forth 语言和栈式机器的历程:既赞叹其简洁优雅、代码量小、可自举的优点,也坦率指出栈操作在实际编程中的痛苦,以及 Forth 社区文化中的问题。
编程与软件工程上首页 8 次 · 2010–2024yosefk.com编辑精选