系统与基础设施multithreaded.stitchfix.com
工程师不该自己写 ETL
Engineers Shouldn’t Write ETL
2016 — 2018，它一次次回到首页
黑点是上首页（≥ 50 分）的那次投稿，空心点是没冲上去的投稿，点越大分数越高。
导读
Stitch Fix 工程博客的文章，主张软件工程师不应亲自编写 ETL（抽取、转换、加载）数据管道，而应把这项工作交给更接近数据的数据科学家或专门角色端到端负责，以免工程师被重复性管道工作消耗，也让数据工作者保有完整自主权。文章由此引出“思考者”与“执行者”之间的分工、功劳归属与激励扭曲问题。
为什么经久不衰：它精准戳中了数据团队里“谁写管道、谁拿功劳”的长期痛点，引发的工程师与数据科学家分工之争至今仍有现实意义。
HN 评论精选
反驳文章把 ETL 工程师等同于平庸，认为喜欢数据工作的人大有人在，这是合理分工而非缺陷。
没人喜欢编写和维护 ETL 数据管道，这是行业里最大的烫手山芋，所以 ETL 岗位成了平庸者的典型温床，这并不奇怪。但认为 ETL 令人灵魂枯竭只是个人观点——有人觉得前端才摧残灵魂，有人就是喜欢处理数据。用 ETL 来判定人平庸，是逻辑不通的。
dizzystar查看英文原文
Nobody enjoys writing and maintaining data pipelines or ETL. It’s the industry’s ultimate hot potato. It really shouldn’t come as a surprise then that ETL engineering roles are the archetypal breeding ground of mediocrity. There is nothing more soul sucking than writing, maintaining, modifying, and supporting ETL to produce data that you yourself never get to use or consume. This is like... your opinion. Some people find pushing around HTML / JS / CSS absolutely soul-crushing. Considering the lion's share of websites are ugly, unusable, and slow, does this mean that front-end engineering is a breeding ground of mediocrity, so server-side devs and CFOs should all be sharing in the pain? Some people actually enjoy working with data, and don't find ETL and pipelining horrible to do at all. It is a different set of challenges, but calling people people mediocre because of ETL is a non-sequitur.
指出思考者与执行者的功劳/责任错配是普遍的组织问题，真正的功劳应归做事者。
思考者/执行者的问题由来已久：在多数组织里，想出点子的人拿大部分功劳，落实想法的人承担大部分工作；若结果不好，思考者总能怪执行不力，把责任推给执行者。这让对“思考”的回报严重失衡，但成功的真正功劳其实应归于实际做事的人。这个问题在每个组织的每个层级都存在——比如想当创业公司商业面（思考者）的人，远比想搞技术的人多。
btilly查看英文原文
The thinker/doer problem goes way back. In most organizations the person who thinks of something gets the lion's share of the credit, and the person who implements it does the lion's share of the work. And if it turns out to be a bad idea, the thinker can always blame a bad implementation, thereby passing the lion's share of the blame to the doer. I've seen careers made and broken based on whether people got to play thinker or doer. This makes rewards for thinking very lopsided. However the problem is that actual credit for success REALLY belongs with the people who did the work. This problem shows up at every scale in every organization. For example there are hundred people who want to be the business side of a startup for every person who wants to build the tech. Why? The business person gets to be the thinker, the developer does the work. And then the business person expects to become the CEO and get the bulk of the payout!
以亲身经历反驳“执行者应当怨恨”的预设：他乐于做管道工程，不想当思考者，分工是个人偏好而非等级。
我在对冲基金做过类似作者所说“数据工程师”的工作：有金融经验的思考者决定导入哪些数据集、如何处理数据里的各种公司行为，我有时提建议，但大多乐于以干净、优雅、稳健、可测试的方式实现他们的想法，也乐于做改进技术栈的管道工作。按作者的说法我该怨恨自己是“执行者/管道工”，但事实相反——我毫无兴趣整天研究彭博手册和符号文件，那知识离开金融业就没用。我喜欢编程、系统设计、构建东西，不觉得这是低贱的工作。
whack查看英文原文
I've worked in a hedge fund in the past, where my role sounded a lot like what the author describes as a "Data Engineer". I would have thinkers, ie people with a lot of financial experience, come up with ideas on which datasets we want to import from which vendors, and how we should handle the 80 different types of corporate actions that are contained within this dataset. I sometimes gave my own suggestions on how to improve upon their ideas, but for the most part, I was happy to focus on implementing their ideas, in the most clean, elegant, robust and testable manner possible. I was happy to do the "plumbing" work of improving upon our tech stack and architecture, in order to make the entire system better functioning and easier to maintain. According to the author, I'm supposed to resent the fact that I'm a "doer/plumber", and not a "thinker". In reality, it was the opposite. Do I really want to spend my entire day reading the Bloomberg manual and figuring out which tables/columns will give us the data we want, and the nuances of what this dataset does and does not cover? Sorry, I have zero interest in doing that. I enjoy programming. I enjoy system design. I enjoy building stuff. I have zero interest in becoming an expert on how to interpret the Bloomberg symbology file. Besides, if I ever left the financial industry and joined a tech company, that knowledge will become completely useless. Did I or anyone consider myself to be a "menial" plumber? I don't think so. I was getting paid hundreds of thousands of dollars, because the "thinkers" recognized the value that I brought to the table. They appreciated that I could quickly and robustly implement the ideas that they had, and keep the system running smoothly without hiccups. They recognized anyone can do a "good enough" job, but it's much much harder to find someone who can do a great job. And for my part, I was perfectly happy to be that guy. If you're someone who wants to expand your breadth and take on more "thinker" responsibilities, more power to you. But just don't forget that there are people like me out there too. There's no shame in being an excellent "doer".
提出用数据科学家加工程师的小团队端到端负责，以问责制替代思考/执行的割裂。
文章诊断得好，但忽略了一个显而易见的解法：组建一个数据科学家加工程师的小团队，端到端负责模型从构建到上线，满足所有 SLA 和性能指标。关键是团队小到可以完全问责，同舟共济，就不必争论谁拿功劳或背锅，而是按最终产品效果来评估。
jimbokun查看英文原文
I think this diagnoses the problem well, but ignores an obvious solution. A team of one data scientist and one engineer, completely responsible for building a model, and seeing it through into production, meeting all applicable SLAs and performance metrics. Or maybe it's two data scientists and one engineer, or one scientist and two engineers, whatever is required. The point is to have a small team you can hold completely accountable for their output. They sink or swim together, so there is no debating whether the scientists or engineers get the credit or take the blame. They are assessed by the effectiveness of the end product they produce.
历次讨论
同类还值得读
TTY 揭秘（2008）
The TTY demystified (2008)
经典文章深入剖析了 Unix/Linux 中 TTY 子系统的工作原理，包括线路规程、终端驱动、信号与控制字符等机制。
系统与基础设施上首页 11 次 · 2009–2025linusakesson.net编辑精选为什么 GNU grep 这么快
Why GNU grep is Fast
FreeBSD 邮件列表上一篇广为流传的帖子，解释 GNU grep 为什么快：核心在于 Boyer-Moore 字符串搜索算法配合精心优化的内部循环和 I/O，让程序在搜索时“…
系统与基础设施上首页 9 次 · 2010–2022lists.freebsd.orgXplain：为普通人讲解 X11
Xplain – Explaining X11 for the rest of us
《Xplain》是作者（HN 用户 Jasper_）写的一系列 X11 长文，用他在浏览器中重建的 X11 绘图模型做交互演示，讲解 X11 的绘图模型与窗口系统设计。
系统与基础设施上首页 6 次 · 2013–2020magcius.github.io编辑精选旋风教程：为 Linux 创建极小 ELF 可执行文件（1999）
A Whirlwind Tutorial on Creating Really Teensy ELF Executables for Linux (1999)
breadbox 写于 1999 年的经典教程，主题是如何为 Linux 手工构造并裁剪出体积极小的 ELF 可执行文件。
系统与基础设施上首页 6 次 · 2007–2025muppetlabs.comPostgreSQL 内部原理
The internals of PostgreSQL
《The Internals of PostgreSQL》是一本免费在线书籍，系统讲解 PostgreSQL 的内部实现，包括查询处理、并发控制、WAL、缓冲区管理等核心机制。
系统与基础设施上首页 5 次 · 2017–2025interdb.jp编辑精选