AI 与机器学习github.com编辑精选
Karpathy 的 MinGPT：极简 GPT 实现
Karpathy's MinGPT
2020 — 2022，它一次次回到首页
黑点是上首页（≥ 50 分）的那次投稿，空心点是没冲上去的投稿，点越大分数越高。
导读
minGPT 是 Andrej Karpathy 写的一个极简 GPT 实现，用少量清晰、集中的 PyTorch 代码演示 GPT 的训练与推理，目标是让人能完整读懂并方便改造。它多次登上 Hacker News，评论围绕其简洁性、可扩展到非语言序列任务，以及社区协作训练大模型的可能性展开。
为什么经久不衰：它把通常分散复杂的 GPT 实现压缩成可完整读懂的代码块，降低了理解门槛，也方便直接修改复用，因此多年后仍值得读。
HN 评论精选
前沿神经网络技术本身并不复杂，复杂度多来自工程加速；minGPT 用少量代码展示了这一点。
我觉得很多人没有意识到，最先进的神经网络技术其实非常简单。能看到这样一份示例——用很少的代码就能得到令人惊叹的结果——非常有价值。你当然可以说它依赖 PyTorch，而 PyTorch 本身代码量很大，但大部分复杂度来自 GPU/TPU/SIMD 加速的需要。一个真正从零开始、只用 CPU 的 C 实现，代码量也不会很大。
modeless查看英文原文
I think many people don't appreciate just how simple state-of-the-art neural net techniques are. It's great to have an illustration of just how little code you need to get the results that have amazed people. You could say that it relies on PyTorch which is a lot of code, but most of the complexity comes from the need to do GPU/TPU/SIMD acceleration. A truly from-scratch CPU-only implementation in C would still not be a large amount of code.
Karpathy 自述写 minGPT 是为了对抗复杂框架，倡导可完整理解、直接改造的代码块。
挺有趣在 HN 上看到这个。它是个比较老的项目，但我一直很喜欢，也仍在维护。当年我想训练一个 GPT，发现现有实现相当复杂、分散在很多文件里，还有太多用于冷门选项的 kwargs 开关，把代码搞臃肿了。但在我脑子里 GPT 是个超级简单、规整的模型，于是我就写了 minGPT。这个项目的影响超出我最初想象，进入了不少项目和论文。我喜欢的是，很多作者会直接在代码里“hack” minGPT，而不是去配置一个包罗万象的 kwarg 怪物。我觉得这有种美感。我常常希望我们有更多 gist、更少框架——看一段代码、完全理解它、按需调整，再在项目里复用，就像细菌交换小 DNA 质粒一样。minGPT 就是为想要这种 GPT 项目方式的人写的。这种方式也有很多缺点，但最终我认为两种方式都有价值。未来 minGPT 的方向是更多示例和更多“牙齿”：应该能在单个多 GPU 节点上用 minGPT 演示训练相对认真的（几 B 参数）模型，并复现该规模的一些基准。
karpathy查看英文原文
Hah funny to see this on HN, it is a relatively old project but one that I continue to love and still work on. I was trying to train a GPT one day and discovered that available implementations were quite complex, spread across many files, and took way too many kwargs switches for esoteric/rare options that just bloated and complexified the code. But in my head a GPT was a super simple neat, isotropic model, so I got all worked up and wrote minGPT. The project went on to have more impact than I originally imagined and made its way into a number of projects and papers. One of those I found only a few days ago here: https://twitter.com/karpathy/status/1566100736076697600 . What I love about these projects is that the authors often "hack up" minGPT in code directly. They don't configure a comprehensive kwarg monster. I think there's a beauty in that. Very often I wish we had more gists and fewer frameworks - to look at code chunks, understand them completely, tune them to our needs, and re-use them in projects, similar to how bacteria trade little DNA plasmids. minGPT is written for those who want that for their GPT projects. There's plenty of cons to this approach too, ultimately I think there's value in both approaches. Coming up the theme of future minGPT development: more examples, and more teeth - it should be possible to demonstrate the training of relatively serious (~few B) models with minGPT on one n-gpu node and reproduce some benchmarks around that scale, but never sacrifice its readability.
minGPT 自包含且易扩展，可作为通用序列建模器用于游戏世界模型、机器人模仿学习等非语言任务。
这其实是个很简洁、自包含的实现，很容易扩展到典型自然语言模型之外，比如为视频游戏创建世界模型，或者创建能从大量混乱的人类演示数据中学习模仿的机器人模型（免责声明：我是第二篇论文的作者之一）。基本上，GPT（或 minGPT）模型是非常出色的序列建模器，几乎可以扔任何合理的序列数据进去，只要不过拟合，就有望得到有趣结果。虽然我只做了大约六年机器学习，但看到这个领域最近变化如此之快还是很震撼，包括扩散模型和 Transformer。可以毫不夸张地说，我们可能在这个十年结束前看到更多重大突破，最终到达现在无法想象的地步。
polygamous_bat查看英文原文
This is actually a pretty neat, self-contained implementation that can super easily extended beyond stereotypical natural language models, for example to create world models for video games [1] or to create robot models that can learn to imitate from large, chaotic human demonstration data [2] (disclaimer, I'm an author on the second one.) Basically, GPT (or minGPT) models are EXCELLENT sequence modelers, almost to the point where you can throw any sensible sequence data at it and hope to get interesting results, as long as you don't overfit. Even though I have only been working on machine learning for around six years, it's crazy to see how the landscape has changed so fast so recently, including diffusion models and transformers. It's not too much to say that we might expect more major breakthroughs by the end of this decade, and end in a place we can't even imagine right now! [1] https://github.com/eloialonso/iris [2] https://github.com/notmahi/bet
提出用分布式闲置 GPU 社区协作训练大模型的设想，成本可能主要只是电费。
我想知道，社区训练模型是否可行？比如让几万名开发者在夜间用 GPU 跑一个类似 SETI@home 的应用，最后大家获得一个训练好的 175B 模型。如果训练成本是 500 万美元，但我印象中那是按云 GPU 价格估算的；如果用闲置算力，你只需要付电费。
martythemaniak查看英文原文
I wonder, is a community trained model feasible? As in, get a few tens of thousands of dev to run a seti@home type app on their GPUs during the night, and at the end you get access to the 175B trained model. If it cost 5m to train, but IIRC that was estimated at cloud gpu prices, if you're using spare capacity you're just paying for electricity.
历次讨论
同类还值得读
如何写一个拼写纠正器
How to Write a Spelling Corrector
Peter Norvig 用一个几十行的 Python 程序演示如何做拼写纠正：先用编辑距离（删除、插入、替换、换位）生成所有可能的候选词，再用从大量文本中统计出的词频概率，选出…
AI 与机器学习上首页 10 次 · 2007–2022norvig.com编辑精选面向黑客的概率编程与贝叶斯方法
Probabilistic Programming and Bayesian Methods for Hackers (2013)
《黑客的贝叶斯方法与概率编程》是一本开源的贝叶斯统计入门书，以 Python/PyMC 代码和交互式 Jupyter Notebook 为主线，强调“先实践、后数学”的学习路径，…
AI 与机器学习上首页 5 次 · 2013–2018camdavidsonpilon.github.io编辑精选神经网络黑客指南
Hacker's Guide to Neural Networks
Andrej Karpathy 写的神经网络入门教程，面向程序员，从最简单的表达式求导讲起，逐步推导出反向传播和梯度下降，最后引出神经网络。
AI 与机器学习上首页 4 次 · 2014–2019karpathy.github.io编辑精选神经网络能逼近任意函数的可视化证明
A visual proof that neural nets can approximate any function
这一章来自 Michael Nielsen 的在线书《Neural Networks and Deep Learning》，用 sigmoid 单元叠加成局部隆起/平滑阶跃的图示…
AI 与机器学习上首页 4 次 · 2014–2022neuralnetworksanddeeplearning.com编辑精选论乔姆斯基与统计学习的两种文化
On Chomsky and the Two Cultures of Statistical Learning
Norvig 回应 Chomsky 对统计语言模型批评的文章，围绕“统计学习能否解释语言”展开，比较生成与解释、少参数模型与 Shannon 式高参数统计模型。
AI 与机器学习上首页 4 次 · 2011–2025norvig.com编辑精选