工具与资源dmst.aueb.gr
dgsh：有向图 Shell
Dgsh – Directed graph shell
2017 — 2025，它一次次回到首页
黑点是上首页（≥ 50 分）的那次投稿，空心点是没冲上去的投稿，点越大分数越高。
导读
dgsh 是一个“有向图 shell”，扩展了传统 Unix 管道，允许用 shell 语法构造多路分支、合并等并行数据流图，而不仅是单线管道。页面上提供了工具介绍和示例，例如用多个工具并行处理数据。
为什么经久不衰：它把 Unix 的管道哲学推广到有向图，多次登上 HN 说明这一思路持续吸引人，今天仍值得研究。
HN 评论精选
提供了一篇相关论文，并指出 dgsh 的设计原语尚需深究。
这看起来很有意思，但我需要更仔细地看示例才能理解为什么选择这些原语（多路管道、多路管道块和存储值）。这里有一篇 2009 年的论文《Composing and executing parallel data-flow graphs with shell pipes》，也是 bash 扩展。我对任何能成功修改 bash 源码的人都感到佩服。该论文模型完全不同，可能更适合“大数据”。
chubot查看英文原文
This looks pretty interesting, although I'll have to dig more into the examples to see why they chose this set of primitives (multipipes, multipipe blocks, and stored values). Here is a 2009 paper, "Composing and executing parallel data-flow graphs with shell pipes", which is also a bash extension. (I'm impressed with anyone who successfully enhances bash's source code.) Although it has a completely different model and I think more suitable for "big data". https://scholar.google.com/scholar?cluster=9869759847871430654&hl=en&as_sdt=0,5&sciodt=0,5 http://dl.acm.org/citation.cfm?id=1645175 In this paper we extend the concept of shell pipes to incorporate forks, joins, cycles, and key-value aggregation. I have a printout of this paper, but unfortunately it doesn't appear to be online :-(
讨论 dgsh 语法与现有 shell 兼容性的问题，并指出它实际是 DAG。
我一直想把这种功能集成到 elvish 里，但想不出好的语法。dgsh 的语法不错，但遗憾的是 & 破坏了其传统语义。有人能想到一个与传统兼容的语法吗？另外吹毛求疵一下，更准确的叫法是有向无环图 shell 或叫 DAG shell，这个语言似乎不允许循环，dagsh 读起来也比 dgsh 顺口。
xiaq查看英文原文
I've always thought about integrating this functionality into elvish https://github.com/elves/elvish but cannot cone up with a good syntax. dgsh has a good one, but unfortunately using & breaks its traditional semantics. Does anyone has some idea of a tradition-compatible grammar? Also, to nitpick, this is more accurately called a directed acyclic graph shell, or simply a DAG shell. The language doesn't seem to allow cycles. dagsh reads nicer than dgsh too.
对工具发明 DSL 持怀疑态度，但肯定 dgsh 复用工具的思路，并推荐 airflow/luigi。
我接触和审视过很多数据处理工具，它们试图帮助构建数据管道，为了性能、可复现性或代码一致性。我发现：大部分发明新语言、或把复杂流程塞进不太合适的语法环境的工具，都不太受欢迎。每当有工具为数据处理发明 DSL，我都会拿出真实世界中丑陋用例清单，多数工具很快就失败了。编程语言本身已经够用，面对脏数据你终究需要一门编程语言。我会试试 dgsh，它复用工具和 UNIX 精神看起来不错。但网站上“C 代码度量”的示例让我想到 awk——虽然优美，但程序超过一百行就该换别的语言了。airflow 和 luigi 这两个 Python 库在数据管道编排上做得很好。
mtrn查看英文原文
I've worked with and looked at a lot of data processing helpers. Tools, that try to help you build data pipelines, for the sake of performance, reproducibility or simply code uniformity. What I found so far: Most tools, that invent a new language or try to cram complex processes into lesser suited syntactical environments are not loved too much. A few people like XSLT, most seem to dislike it, although it has a nice functional core hidden under a syntax that seems to come from a time, where the answer to everything was XML. There are big data orchestration frameworks, that use an XML as configuration language, which can be ok, if you have clear processing steps. Every time a tool invents a DSL for data processing, I grab my list of ugly real world use cases and most of the tools fail soon, if not immediately. That's a pity. Programming languages can be effective as they are, and with the exceptions that unclean data brings, you want to have a programming language at your disposal anyway. I'll give dgsh a try. The tool reuse approach and the UNIX spirit seems nice. But my initial impression of the "C code metrics" example from the site is mixed: It reminds me of awk, about which one of the authors said, that it's a beautiful language, but if your programs getting longer than hundred lines, you might want to switch to something else. Two libraries which have a great grip at the plumbing aspect of data processing systems are airflow and luigi. They are python libraries and with it you have a concise syntax and basically all python libraries plus non-python tools with a command line interface at you fingertips. I am curious, what kind of process orchestration tools people use and can recommend?
提出 ETL 工具需要事务性错误处理，这是健壮数据管道的缺失项。
这可能有点偏题，但我真心希望更多数据处理/ETL 工具支持“事务单元”的概念。太多工具一开始就认为“要尽可能多地塞进独立数据块”。稳健系统真正需要的往往是对错误处理的可靠支持，例如“如果这块数据没进去，那块也不该进去”。数据总是又脏又乱，许多 ETL 系统在设计上并没有考虑这一点。当然，也许我只是没找到合适的工具——有人知道特别擅于此道的工具吗？
karlmdavis查看英文原文
This is perhaps a bit off-topic, but what I really wish more data processing/ETL tools supported is the concept of transactional units. Too many of them seem to start with the worldview that "we need to shove in as many of the separate bits as we possibly can." What's often needed for robust systems, instead, is solid support for error handling such that "if this bit doesn't make it in, then neither does that bit." Data is always messy and dirty, and too many ETL systems don't seem architected to cope with that reality. Of course, maybe I just haven't found the right tools. Anyone know of tools that handle this particularly well?
历次讨论
同类还值得读
直接在浏览器中启动 Linux 内核
Boot a linux kernel right inside your browser.
Fabrice Bellard 的 JSLinux 页面，用 JavaScript 在浏览器里启动一个 Linux 内核，并提供一个可交互的终端环境。
工具与资源上首页 5 次 · 2011–2025bellard.orgHacker News 未公开功能与行为清单（2018）
A List of Hacker News's Undocumented Features and Behaviors (2018)
数据科学家 Max Woolf 整理的 Hacker News 未公开功能与行为清单，涵盖投票机制、排名算法、账号惩罚等社区运作细节。
工具与资源上首页 5 次 · 2019–2025github.comMapSCII：在终端中用盲文和 ASCII 渲染地图
MapSCII – Braille and ASCII map renderer for the console
MapSCII 是一个在命令行终端里渲染地图的开源工具，用盲文字符和 ASCII 字符把地理数据画成可交互浏览的字符地图。
工具与资源上首页 5 次 · 2017–2025github.com顶尖大学的免费在线课程
Free Online Courses from Top Universities
Open Culture 整理的一份免费在线课程合集，汇集了全球顶尖大学（如 MIT、斯坦福、耶鲁等）向公众开放的网络课程资源，涵盖音频、视频等多种形式。
工具与资源上首页 5 次 · 2010–2022openculture.com淘气字符串：容易引发用户输入问题的字符串清单
Naughty Strings: A list of strings likely to cause issues as user-input data
一个 GitHub 仓库，收录了大量容易让程序出问题的用户输入字符串，如特殊字符、emoji、SQL 注入片段、系统命令等，供开发者用于测试输入处理逻辑。
工具与资源上首页 4 次 · 2015–2020github.com