算法与计算机科学stevehanov.ca
20 行代码每次都击败 A/B 测试
20 lines of code that beat A/B testing every time
2012 — 2025，它一次次回到首页
黑点是上首页（≥ 50 分）的那次投稿，空心点是没冲上去的投稿，点越大分数越高。
导读
这篇文章用约 20 行代码实现一个简单的 epsilon-greedy 多臂老虎机：不再固定分流做 A/B 测试，而是根据实时转化表现，逐步把更多流量分配给当前更优的版本。作者主张这种自适应分配能更快收敛，并减少把流量浪费在差版本上的损失；标题里的“每次都能击败 A/B 测试”是吸引眼球的说法。
为什么经久不衰：它用极短代码把多臂老虎机中的“探索与利用”权衡讲得可操作，而评论区又留下了关于非平稳流量、交互效应和指标噪声的经典讨论，因此多年后仍值得一读。
HN 评论精选
指出非平稳流量，以及测试与独立改动之间的交互效应，会让 bandit 结果严重偏斜，而规范 A/B 测试没有这些问题。
这个思路发人深省，但方法有显著问题。真实世界的表现会随时间变化，比如转化率有日、周、月波动；A/B 测试不受影响，而这种方法如果随机切换方向时恰好赶上转化率向好的波动，就会误判。更关键的是，它会制造测试与独立改动之间长期存在的交互效应：假设 A 略好，但做网站重大改进时 B 暂时略好，新流量就会更多给 B，B 凭这波好流量压过 A；这种相关性会持续到 A 的新流量完全替换旧流量，若 A 只有 5% 流量可能要等极久。规范构造的 A/B 测试没有这个统计异常。
btilly查看英文原文
This is thought-provoking, which is good. However there are significant issues with the approach. 1. Real world performance varies over time. For instance there are typically daily, weekly and monthly conversion rate fluctuations. Not an issue for A/B testing, but a big issue for this approach if a random switch in direction happens at the same time that conversion fluctuations happen to head in a good direction. 2. (This is really a special case of #1 - but a very, very important special case.) This approach creates long-lasting interaction effects between tests and independent changes. That requires explanation. Suppose you're running a test. Version A is somewhat better. But version B is temporarily looking slightly better when you make a significant improvement to your website (maybe you started another test that works). Now you're adding a lot of good traffic to version B (good because of the other change) and very little of the new good traffic to version A. This new version B traffic soundly beats your old version A traffic. This correlation between time and website performance will continue until the old version A traffic is completely swamped by new version A traffic. With only 5% of your traffic going to version A, this can easily take 100x as long as your test has been running - or more. (Properly constructed A/B tests do not suffer this statistical anomaly.) 3. Code over time gets messy. One of the most important characteristics of A/B testing is that you can delete the mess and move on. With this approach you can't - it just hangs around adding to your technical debt. 4. Businesses are complex, and often have multiple measures they would like to balance. For instance in a recent test, conversion to click was hurt, conversion to a person who clicked 5x was helped. A/B testing let us notice that something weird was going on and think about what we really cared about. This automated approach would make a decision and could have hidden a real problem. 5. Many tests perform differently on existing users and new users. A/B testing with proper cohort analysis can let you tease this out and decide accordingly. This approach doesn't give you that kind of sophistication.
历史热度会积累成巨大样本优势，使后来更优的设计难以翻身；可考虑给旧数据加时间衰减。
这个技术有趣，但也有缺陷。如果应用有一段热度很高的时期，当时最受欢迎的设计会获得高点击率和数万样本；等热度过去再推出更好的新设计，它当前表现虽更好，但新增的几百个样本远不足以抵消之前的数万样本，可能很久都追不上。看指标时要理解它由什么构成、可能在哪里出错。一个可能的解法是加衰减因子，让旧数据权重更低。
rauljara查看英文原文
This is an interesting technique, but it too has flaws. If there is a period of buzz and excitement surrounding your app, whatever design was most popular at that time will be rewarded accordingly, and accrue a high click through rate with tens of thousands of case. If you introduce a new superior design after the period of buzz has gone away, the new design may take a very long time to catch up. Even though it is currently outperforming the old, the few hundred cases that are being added won't be enough to offset the tens of thousands that came before. With all metrics, it's important to understand what's actually going into the measure and where it might get tripped up. A potential solution might be to add a decay factor, so that the older data carries less weight.
认为 bandit 的思路正确，但实际更该用结合用户上下文的 contextual bandit。
这是对 A/B 测试最重要的批评，远超过同时推断和 Bonferroni 校正之类老生常谈。Epsilon-greedy 在 k 臂老虎机上表现不错，但多数应用里按用户定制策略会更好，也就是上下文老虎机，已有简单策略效果不错。
equark查看英文原文
This is the most important critique of A/B testing. It far outweighs the traditional hoopla about simultaneous inference and Bonferonni corrections. Epsilon greedy does well on k-armed bandit problems, but in most applications you likely can do significantly better by customizing the strategy to individual users. That's a contextual bandit and there are simple strategies that to pretty well here too. For instance: http://hunch.net/?p=298 http://hunch.net/~exploration_learning/main.pdf http://web.mit.edu/hauser/www/Papers/Hauser_Urban_Liberali_Braun_Website_Morphing_May_2008.pdf
bandit 更适合优化大量内容/关卡，让系统筛选质量，从而解放创作者的产量。
这里大家忽略的一点：不要用 bandit 去 A/B 测试 UI 元素，而要用它优化内容或手游关卡。我的应用用 bandit 优化内容，拿到稳定五星和 100+ 评价。系统自动筛掉差的，我就可以不管质量先大量产出内容，放下完美主义去创作。还提到《思考，快与慢》里的陶艺课研究：按总重量评分的一组在创造力上完胜只交一件作品的一组。
orasis查看英文原文
Here's what everyone is missing. Don't use bandits to A/B test UI elements, use them to optimize your content / mobile game levels. My app, 7 Second Meditation, is solid 5 stars, 100+ reviews because I use bandits to optimize my content. By having the system automatically separate the wheat from the chaff, I am free to just spew out content regardless of its quality. This allows me to let go of perfectionism and just create. There is an interesting study featured in "Thinking Fast and Slow" where they had two groups in a pottery class. The first group would have the entirety of their grade based on the creativity of a single piece they submit. The second group was graded on only the total number of pounds of clay they threw. The second group crushed the first group in terms of creativity.
历次讨论
同类还值得读
位运算技巧集
Bit Twiddling Hacks
斯坦福页面上的经典位运算技巧合集，收录了 popcount、前导/尾随零、位反转、交换、舍入等大量技巧，并附有解释和性能讨论。
算法与计算机科学上首页 8 次 · 2011–2024graphics.stanford.edu编辑精选如何不按平均评分排序
How Not To Sort By Average Rating
经典文章解释了为什么按平均评分排序不可靠：只有少量评价的条目（比如仅一条五星）会轻易排到最前，而评价数量的差异被忽略。
算法与计算机科学上首页 7 次 · 2009–2024evanmiller.org编辑精选数据结构可视化
Data Structure Visualizations
旧金山大学 David Galles 教授制作的数据结构与算法交互式可视化网站，涵盖 B 树、红黑树、排序、图算法等，可逐步操作观察其运行过程。
算法与计算机科学上首页 6 次 · 2011–2021cs.usfca.edu编辑精选1996 年以来的计算机科学最佳论文奖
Best Paper Awards in Computer Science since 1996
Jeff Huang 维护的一个汇总页面，收录了自 1996 年以来计算机科学各主要会议的“最佳论文奖”获奖名单，并按年份和会议整理，方便读者一站式浏览历届获奖论文。
算法与计算机科学上首页 6 次 · 2010–2018jeffhuang.com编辑精选算法（Jeff Erickson）
Algorithms, by Jeff Erickson
伊利诺伊大学 Jeff Erickson 教授的算法课教材，免费在线开放。
算法与计算机科学上首页 5 次 · 2018–2022jeffe.cs.illinois.edu编辑精选