AI 与机器学习arxiv.org
为什么树模型在表格数据上仍优于深度学习？
Why do tree-based models still outperform deep learning on tabular data?
2022 — 2024，它一次次回到首页
黑点是上首页（≥ 50 分）的那次投稿，空心点是没冲上去的投稿，点越大分数越高。
导读
这篇论文通过基准实验比较树模型与深度学习在表格数据上的表现，探讨前者为何仍然占优。作者在约1万样本的中等规模数据上发现树模型保持领先，且这一结论尚未计入其速度优势；附录还初步观察到，增大训练集时神经网络与树模型的差距有缩小趋势。
为什么经久不衰：它系统性地提醒人们：在表格数据上，树模型仍是难以击败的强基线，而不是默认选择深度学习。
HN 评论精选
树模型的优势在于少特征工程、易正则化、当天可用，深度学习需长期调优才可能胜出。
我有一个理论：基于树的模型几乎不需要特征工程。它们能以有原则的方式处理类别数据，能同样轻松地处理最偏态、多峰、异方差的连续数值数据，就像处理0-1缩放的正态分布一样；而且相比深度学习模型（可能有无数参数组合，更别说训练到全局最优），它们更容易正则化。我认为如果你花几个月把数据和模型结构调好，当然可以让深度学习模型超过梯度提升树。但何必呢？梯度提升树今天就能做完。
CapmCrackaWaka查看英文原文
I have a theory - tree based models require minimal feature engineering. They are capable of handling categorical data in principled ways, they can handle the most skewed/multimodal/heteroskedastic continuous numeric data just as easily as a 0-1 scaled normal distribution, and they are easy to regularize compared to a DL model (which could have untold millions of possible parameter combinations, let alone getting the thing to train to a global optimum). I think if you spent months getting your data and model structure to a good place, you could certainly get a DL model to out-perform a gradient boosted tree. But why do that, when the GBT will be done today?
1万样本不算中等，标题可能夸大；附录显示扩大数据时差距缩小但证据有限。
“结果显示，基于树的模型在中等规模数据（约1万样本）上仍是最先进，即使不考虑其明显更快的速度。”这真的算“中等”吗？对我来说这似乎非常小。MNIST有6万样本，ImageNet有数百万。我认为标题夸大了发现。我很有兴趣了解这些方法在更大数据集上如何比较。是否存在一个阈值，深度学习会超过树模型？编辑：他们在附录中提到了这一点：A.2.2 大规模数据集。我们将基准扩展到大规模数据集，在同一组数据集上比较大/中规模设置（训练集截断到5万 vs 1万样本）。我们只保留超过5万样本的数据集并将训练集限制为5万样本。不幸的是这排除了很多数据集，使比较不那么清晰。然而在大多数情况下，增加训练集规模会缩小神经网络与树模型之间的差距。我们将这一趋势的严格研究留待未来工作。
Permit查看英文原文
> Results show that tree-based models remain state-of-the-art on medium-sized data (∼10K samples) even without accounting for their superior speed. Is that really "medium"? That seems very small to me. MNIST has 60,000 samples and ImageNet has millions. I think the title overstates the findings. I'd be interested to hear how these methods compare on much larger datasets. Is there a threshold at which deep learning outperforms tree-based models? Edit: They touch on this in the appendix: > A.2.2 Large-sized datasets > We extend our benchmark to large-scale datasets: in Figures 9, 10, 11 and 12, we compare the results of our models on the same set of datasets, in large-size (train set truncated to 50,000 samples) and medium-size (train set truncated to 10,000 samples) settings. > We only keep datasets with more than 50,000 samples and restrict the train set size to 50,000 samples (vs 10,000 samples for the medium-sized benchmark). Unfortunately, this excludes a lot of datasets, which makes the comparison less clear. However, it seems that, in most cases, increasing the train set size reduces the gap between neural networks and tree-based models. We leave a rigorous study of this trend to future work.
树模型研究投入不足，超参与停止准则仍粗糙，部分因深度学习更受关注且GBM优化需底层C++。
令我困惑的是，考虑到基于树的方法如此有效，研究界对改进它们的关注却如此之少。例如 LightGBM 和 XGBoost 允许一些正则化项，但方差/偏差仍主要靠全局设置最大深度和最大节点数（然后参数搜索找好设置）来控制。肯定有比计算节点数更强大、更精细的方法来决定每棵树何时停止生长？如果这是神经网络，会有上百篇竞争论文提出不同方法并争论优缺点。我不确定问题是不是神经网络天生更吸引人，还是要在GBM上取得SOTA改进必须深入棘手的C++。可能两者都有。
_pastel查看英文原文
It's baffling to me how little research attention there has been to improving tree-based methods, considering their effectiveness. For example, LightGBM and XGBoost allow some regularization terms, but the variance/bias is still mostly controlled by globally setting the max depth and max node count (and then parameter searching to find good settings). Surely there must be more powerful and sophisticated ways of deciding when to stop building each tree than counting the number of nodes? If this was neural nets there would be a hundred competing papers proposing different methods and arguing over their strengths and weaknesses. I'm not sure whether the problem is that neural nets are just fundamentally more sexy, or that in order to make SOTA improvements in GBMs you need to dive into some gnarly C++. Probably both.
提供树模型交互教程，并提醒良好正则化的神经网络也可在表格数据上表现好。
如果你对树模型如何工作感兴趣，我写了决策树的交互式解释（mlu-explain.github.io/decision-tree/）和随机森林的交互式解释（mlu-explain.github.io/random-forest/）。另外值得注意，一篇较新的论文表明，若正则化得当，神经网络在表格数据上也能表现良好（arxiv.org/abs/2106.11189v1）。
jwilber查看英文原文
If you’re interested in how tree-based models work, I wrote an interactive explanation on decision trees here: https://mlu-explain.github.io/decision-tree/ and random forests here: https://mlu-explain.github.io/random-forest/ It’s also worth noting that a recentish paper shows neural networks can perform well on tabular data if well-regularized: https://arxiv.org/abs/2106.11189v1?utm_source=jesper&utm_medium=email&utm_campaign=coming-up-with-weather-puns-is-a-breeze
历次讨论
同类还值得读
如何写一个拼写纠正器
How to Write a Spelling Corrector
Peter Norvig 用一个几十行的 Python 程序演示如何做拼写纠正：先用编辑距离（删除、插入、替换、换位）生成所有可能的候选词，再用从大量文本中统计出的词频概率，选出…
AI 与机器学习上首页 10 次 · 2007–2022norvig.com编辑精选面向黑客的概率编程与贝叶斯方法
Probabilistic Programming and Bayesian Methods for Hackers (2013)
《黑客的贝叶斯方法与概率编程》是一本开源的贝叶斯统计入门书，以 Python/PyMC 代码和交互式 Jupyter Notebook 为主线，强调“先实践、后数学”的学习路径，…
AI 与机器学习上首页 5 次 · 2013–2018camdavidsonpilon.github.io编辑精选神经网络黑客指南
Hacker's Guide to Neural Networks
Andrej Karpathy 写的神经网络入门教程，面向程序员，从最简单的表达式求导讲起，逐步推导出反向传播和梯度下降，最后引出神经网络。
AI 与机器学习上首页 4 次 · 2014–2019karpathy.github.io编辑精选神经网络能逼近任意函数的可视化证明
A visual proof that neural nets can approximate any function
这一章来自 Michael Nielsen 的在线书《Neural Networks and Deep Learning》，用 sigmoid 单元叠加成局部隆起/平滑阶跃的图示…
AI 与机器学习上首页 4 次 · 2014–2022neuralnetworksanddeeplearning.com编辑精选论乔姆斯基与统计学习的两种文化
On Chomsky and the Two Cultures of Statistical Learning
Norvig 回应 Chomsky 对统计语言模型批评的文章，围绕“统计学习能否解释语言”展开，比较生成与解释、少参数模型与 Shannon 式高参数统计模型。
AI 与机器学习上首页 4 次 · 2011–2025norvig.com编辑精选