AI 与机器学习elijahpotter.dev
马尔可夫链是最早的语言模型
Markov chains are the original language models
2024 — 2025，它一次次回到首页
黑点是上首页（≥ 50 分）的那次投稿，空心点是没冲上去的投稿，点越大分数越高。
导读
这篇文章介绍马尔可夫链——通过统计前文来预测下一个词的简单模型——并指出它其实是语言模型最早的雏形。作者回顾了用马尔可夫链生成文本的原理与局限，并借此帮助读者理解现代大语言模型（LLM）的基本工作方式。
为什么经久不衰：它用最简单的“下一个词预测”模型讲清了 LLM 的核心直觉，是理解大模型为何会“幻觉”、擅长什么的好入门，因此多年反复被推荐。
HN 评论精选
马尔可夫链的线性状态本质使其无法利用远距离上下文，这正是注意力机制出现的原因
问题在于马尔可夫链的线性本质：观察之后你就绝对处于一个新状态，A 到 B 到 C。一个经典例子是输入二维位图：图案是纵向的，但数据按从左到右传入，马尔可夫链只依赖当前的横向输入，完全捕捉不到这些模式。语言也类似——语言不是线性的，几句话之前的上下文应该改变当前字符序列的概率。注意力机制是我们目前解决这个问题的最佳方案，而马尔可夫链只能勉强串起几个音节。我玩过很多马尔可夫链，试过跳跃状态等技巧，但最终总是被推向类似注意力机制的做法来处理上下文。
AnotherGoodName查看英文原文
The problem is the linear nature of markov chains. Sure they can branch but after an observation you are absolutely at a new state. A goes to B goes to C etc. A classic problem to understand why this is an issue is feeding in a 2D bitmap where the patterns are vertical but you’re passing in data left to right which Markov chains can’t handle since they are navigating exclusively on the current left to right inout. They miss the patterns completely. Similar things happen with language. Language is not linear and context from a few sentences ago should change probabilities in the current sequence of characters. The attention mechanism is the best we have for this and Markov chains struggle beyond stringing together a few syllables. I have played with Markov chains a lot. I tried having skip states and such but ultimately you’re always pushed towards doing something similar to the attention mechanism to handle context.
把 LLM 简化理解为“超大号马尔可夫链”是普通人可用的心智模型
多年来我一直告诉别人：对大语言模型一个合理可行的、简化的心智模型，就是一个接入了无限大加权语料库的马尔可夫链生成器。懂 LLM 的人很少反驳这个说法，最多说它粗糙、弱化了细节。既然开头就说了是简化，我认为这个比喻是成立的。
cestith查看英文原文
I’ve been telling people for years that a reasonably workable initial, simplified mental model of a large language model is a Markov chain generator with an unlimited, weighted corpus trained in. Very few people who know LLMs have said anything to critique that thought more than that it’s a coarse description and downplays the details. Since being simplified is in the initial statement and it’s not meant to capture detail, I say if it walks like a really big duck and it honks instead of quacking then it’s maybe a goose or swan which are both pretty duck-like birds.
马尔可夫链虽远比 LLM 简单，却是向非技术人群解释 LLM 原理的最佳教学工具
LLM 内部实际发生的事情比马尔可夫链复杂许多个数量级。但我同意它们是讲解 LLM 基本原理的绝佳教学工具，甚至对非技术听众也有效。很多人从神经网络原理开始“解释”LLM，效果往往不好，因为需要很大的概念跳跃。而解释“LLM 本质上就是基于前文的统计模型反复做下一个词预测”，大多数人都能理解，并且有助于建立直觉：为什么模型会幻觉、擅长什么。马尔可夫链是最简单的下一个词预测模型，15 分钟就能讲明白，是向普通人解释 LLM 的好方式。
lukev查看英文原文
What's actually happening in a LLM is many orders of magnitude more complex than a Markov chain. However, I agree that they're an amazing pedagogical tool for the basic principles of how a LLM works, even to a non-technical audience. Many people try to "explain" LLMs starting with the principles of neural networks. This rarely works well: there are some significant conceptual leaps required. However, explaining that a LLMs are really just iterated next-word prediction based on a statistical model of the preceding words is something that most people can grok, and in a useful way: in my experience, it actually helps give people a useful intuition for why and how models hallucinate and what kind of things they're good/bad at. Markov chains are super simple iterated next-word prediction model, and can be explained in 15 minutes. It's a great way to explain LLMs to laypeople.
马尔可夫链是入门“让代码从数据中学习”的最简单实践项目
马尔可夫链的能力确实很有限，但我喜欢它的一点是：它是让代码“从数据中学习”的最简单、最直观的方式之一。如果你从没写过会学习的东西，试试吧！这是我最近写的一个很初级的示例，边做边讲解基本思想——从生成美国城市名开始，最后基于《龙与地下城 DM 指南》生成文本。
unoti查看英文原文
It's true that Markov chains are very limited in their capabilities. But one thing I love about them is that they are one of the simplest and most intuitive ways to write code that *learns* from input data. If you're never written something that *learns*, try it out! Here's a very primitive one I wrote recently to explain the basic idea and explains it along the way. https://github.com/unoti/markov-basics/blob/main/markov-basics.ipynb This starts with generating US city names, and ends with generating text based on the Dungeons and Dragons DM guide.
历次讨论
同类还值得读
如何写一个拼写纠正器
How to Write a Spelling Corrector
Peter Norvig 用一个几十行的 Python 程序演示如何做拼写纠正：先用编辑距离（删除、插入、替换、换位）生成所有可能的候选词，再用从大量文本中统计出的词频概率，选出…
AI 与机器学习上首页 10 次 · 2007–2022norvig.com编辑精选面向黑客的概率编程与贝叶斯方法
Probabilistic Programming and Bayesian Methods for Hackers (2013)
《黑客的贝叶斯方法与概率编程》是一本开源的贝叶斯统计入门书，以 Python/PyMC 代码和交互式 Jupyter Notebook 为主线，强调“先实践、后数学”的学习路径，…
AI 与机器学习上首页 5 次 · 2013–2018camdavidsonpilon.github.io编辑精选神经网络黑客指南
Hacker's Guide to Neural Networks
Andrej Karpathy 写的神经网络入门教程，面向程序员，从最简单的表达式求导讲起，逐步推导出反向传播和梯度下降，最后引出神经网络。
AI 与机器学习上首页 4 次 · 2014–2019karpathy.github.io编辑精选神经网络能逼近任意函数的可视化证明
A visual proof that neural nets can approximate any function
这一章来自 Michael Nielsen 的在线书《Neural Networks and Deep Learning》，用 sigmoid 单元叠加成局部隆起/平滑阶跃的图示…
AI 与机器学习上首页 4 次 · 2014–2022neuralnetworksanddeeplearning.com编辑精选论乔姆斯基与统计学习的两种文化
On Chomsky and the Two Cultures of Statistical Learning
Norvig 回应 Chomsky 对统计语言模型批评的文章，围绕“统计学习能否解释语言”展开，比较生成与解释、少参数模型与 Shannon 式高参数统计模型。
AI 与机器学习上首页 4 次 · 2011–2025norvig.com编辑精选