AI 与机器学习karpathy.github.io编辑精选
循环神经网络令人意外的有效性
The Unreasonable Effectiveness of Recurrent Neural Networks
2015 — 2019，它一次次回到首页
黑点是上首页（≥ 50 分）的那次投稿，空心点是没冲上去的投稿，点越大分数越高。
导读
Karpathy 2015 年的博客长文：用字符级循环神经网络（char-rnn）在原始文本上训练——莎士比亚剧本、LaTeX 论文、Linux 源码、婴儿名字等——展示模型竟能学会引号配对、括号闭合、缩进与代码结构等出人意料的规律。文中给出极简实现思路（后来常被引用的 112 行 Python 版本），并讨论了 RNN 的能力边界与训练细节。
为什么经久不衰：它用可复现的小实验把「简单模型也能学到惊人结构」讲得清清楚楚，是理解序列建模的经典入门，也是 2015 年前后深度学习氛围的历史坐标——如今读来还能对照 RNN 被 Transformer 取代的这十年。
HN 评论精选
称赞 Karpathy 深入浅出地科普深度学习，并预言模型堆叠/多任务学习会成为常见做法。
Karpathy 是我最喜欢的作者之一——他不仅深度参与技术工作（可以去看 CS231n 课程），还花大量时间把整个领域讲得通俗易懂，这是鼓励别人探索这个领域的好办法。如果你喜欢他的博客，强烈推荐看他的「用 ConvNet 和 RNN 做自动图像描述」演讲，里面提到很多他还没在文章里展开的观点。他很谦虚地说自己的工作只是把图像识别（CNN）叠加到句子生成（RNN）上，让梯度促使两者协同。既然现在机器足够强大，我认为会看到大量把原本分离的模型堆叠起来以提升性能或做多任务学习的做法。
Smerity查看英文原文
Karpathy is one of my favourite authors - not only is he deeply involved in technical work (audit the CS231n course for more[1]!), he spends much of his time demystifying the field itself, which is a brilliant way to encourage others to explore it :) If you enjoyed his blog posts, I highly recommend watching his talk on "Automated Image Captioning with ConvNets and Recurrent Nets"[2]. In it he raises many interesting points that he hasn't had a chance to get around to fully in his articles. He humbly says that his captioning work is just stacking image recognition (CNN) on to sentence generation (RNN), with the gradients effectively influencing the two to work together. Given that we've powerful enough machines now, I think we'll be seeing a lot of stacking of previously separate models, either to improve performance or to perform multi-task learning[3]. A very simple concept but one that can still be applied to many other fields of interest. [1]: http://cs231n.stanford.edu/ [2]: https://www.youtube.com/watch?v=xKt21ucdBY0 [3]: One of the earliest - "Parsing Natural Scenes and Natural Language with Recursive Neural Networks" http://nlp.stanford.edu/pubs/SocherLinNgManning_ICML2011.pdf
2019 年的回望：卷积与 Transformer 已在序列任务上全面超越 RNN。
如果这篇文章写于今天，Karpathy 恐怕得把它叫作「卷积的不可思议的有效性」。自 2015 年以来，卷积、因果卷积和空洞卷积，尤其是带注意力的卷积（如 Transformer），大举侵占了 RNN 的地盘，如今在大多数（也许全部）序列相关任务上都是最优。看来 RNN 就是没能好好利用循环和隐状态记忆，而卷积的非局部性和易优化特性让训练更快、模型更大，从而效果更好。谁能想到呢？
gwern查看英文原文
If this were written today, Karpathy would have to call it "The Unreasonable Effectiveness of Convolutions". Since 2015, convolutions, causal or dilated convolutions, and especially convolutions with attention like the Transformer, have made remarkable inroads onto RNN territory and are now SOTA for most (all?) sequence-related tasks. Apparently RNNs just don't make very good use of that recurrency & hidden memory, and the non-locality & easy optimization of convolutions allow for much better performance through faster training & bigger models. Who knew?
实践者补充：封装后的 char-rnn 更好用，LSTM 在文本生成上仍很强。
我做了 textgenrnn，作为创建 char-rnn 的更上层封装，解决了冷启动问题（不需要种子文本），并加入了一些 2015 年之后的新发现，比如注意力机制和 CuDNN 加速。处理通用文本分类问题还有其他思路（比如 BERT），但就文本生成而言，LSTM 依然难以被击败，尽管它在长程依赖上仍有问题。
minimaxir查看英文原文
I made textgenrnn (https://github.com/minimaxir/textgenrnn) as a higher-level approach to creating char-rnns, solving some issues such as the cold start problem (textgenrnn doesn't need a seed) and incorporating a few newer discoveries since 2015 such as Attention and CuDNN speedups. There are other strategies for working with text to solve generic classification problems (e.g. BERT), but for text generation, LSTMs still can't be beat even though it still has issues with longer-term dependencies.
强调极简实现即可见效，且远胜 HMM 等传统方法。
RNN 最酷的地方就是它们有多简单。这 112 行 Python 代码就足以生成相当逼真的莎士比亚式散文（或者特朗普推文，如果你好这口）。效果比用 HMM 好得多。
bjourne查看英文原文
The cool part of RNNs are how simple they are. https://gist.github.com/karpathy/d4dee566867f8291f086 These 112 lines of Python is all it takes to very realistic Shakespeare prose (or Trump tweets if that's your thing). The results are much better than when using HMMs.
历次讨论
同类还值得读
如何写一个拼写纠正器
How to Write a Spelling Corrector
Peter Norvig 用一个几十行的 Python 程序演示如何做拼写纠正：先用编辑距离（删除、插入、替换、换位）生成所有可能的候选词，再用从大量文本中统计出的词频概率，选出…
AI 与机器学习上首页 10 次 · 2007–2022norvig.com编辑精选面向黑客的概率编程与贝叶斯方法
Probabilistic Programming and Bayesian Methods for Hackers (2013)
《黑客的贝叶斯方法与概率编程》是一本开源的贝叶斯统计入门书，以 Python/PyMC 代码和交互式 Jupyter Notebook 为主线，强调“先实践、后数学”的学习路径，…
AI 与机器学习上首页 5 次 · 2013–2018camdavidsonpilon.github.io编辑精选神经网络黑客指南
Hacker's Guide to Neural Networks
Andrej Karpathy 写的神经网络入门教程，面向程序员，从最简单的表达式求导讲起，逐步推导出反向传播和梯度下降，最后引出神经网络。
AI 与机器学习上首页 4 次 · 2014–2019karpathy.github.io编辑精选神经网络能逼近任意函数的可视化证明
A visual proof that neural nets can approximate any function
这一章来自 Michael Nielsen 的在线书《Neural Networks and Deep Learning》，用 sigmoid 单元叠加成局部隆起/平滑阶跃的图示…
AI 与机器学习上首页 4 次 · 2014–2022neuralnetworksanddeeplearning.com编辑精选论乔姆斯基与统计学习的两种文化
On Chomsky and the Two Cultures of Statistical Learning
Norvig 回应 Chomsky 对统计语言模型批评的文章，围绕“统计学习能否解释语言”展开，比较生成与解释、少参数模型与 Shannon 式高参数统计模型。
AI 与机器学习上首页 4 次 · 2011–2025norvig.com编辑精选