编程与软件工程instagram-engineering.com
Instagram 工程实践:大规模 Python 的严格模块化
Python at Scale: Strict Modules
2019 — 2020,它一次次回到首页
黑点是上首页(≥ 50 分)的那次投稿,空心点是没冲上去的投稿,点越大分数越高。
导读
Instagram 工程团队介绍其自研的“严格模块”(Strict Modules)静态检查机制:通过禁止在模块导入时执行 I/O、修改全局状态等副作用,来加速启动、避免导入即产生隐式行为导致的生产事故。文章展示了大型 Python 代码库如何在保持动态语言灵活性的同时引入更严格的约束。
为什么经久不衰:它展示了超大规模 Python 团队如何用静态分析驯服“导入即执行副作用”这一顽疾,其思路对今天任何大型 Python 项目仍有借鉴意义。
HN 评论精选
导入即产生副作用(如隐式建数据库连接)是 Python 生态的痛点,应把初始化显式放进 setup 函数
> 这意味着仅仅导入这个模块,我们就在别处修改了全局状态。没错,就是这样!这正是我最讨厌 Django 和某些 Flask 应用的原因:导入一个模块就会隐式地建立数据库连接、做一堆其他魔法操作,导致我没法在 `python manage.py` 之外导入该模块里定义的常量。而且如文章所说,这让“数据库暂时不可用”的处理变得困难得多(因为有人把建立连接的代码放在了某个模块的全局作用域里)。我更喜欢那些代码只在调用其 setup 函数后才执行的框架/模块。
allan_s查看英文原文
> This means that just by importing this module, we're mutating global state somewhere else. Yes, this ! That's why I hate Django and some flask app the most for, the fact that by importing a module, you're implicitly creating a database connection, and a lot of other magic stuff, which mean that now I can't import a constant defined in said module outside of `python manage.py` Also as said below in the article, suddenly it's much harder to handle smoothly the "the database is momentary unavailable" (because someone has put the line starting the database connection in the global space of a module somewhere) I much prefer frameworks/modules for which code is executed only once you invoke their "setup" function
希望出现一门内置类型检查、可编译的 Python 严格子集语言,去掉阻碍优化的过度动态特性
我越来越希望有人创造一门新语言:它是 Python 的严格子集,内置 mypy,并能编译成机器码。Python 的语法、社区是我最喜欢的,生产力也最高。只是它有太多过度动态的特性——大多数人根本不用,但用得又刚好够多,足以阻碍优化和结构性改进。
timothycrosley查看英文原文
More and more I want someone to create a new language that amounts to a strict subset of Python, with mypy built-in, and is compilable into machine code. Python has by far my favorite syntax, community, and in my experience leads to the greatest productivity. There just happens to be a lot of overly dynamic features, that aren't even used by most, but used just enough to hold back optimization and structural improvement.
希望看到实际工具发布和启动时间收益的量化数据;分享了自己团队用 Lazy 包装器避免模块级 I/O 的做法
我喜欢这个想法,但目前感觉还只是个想法。我更想看到他们发布编译期分析工具,并公布到底节省了多少启动时间的实测数据。在我们的代码库里,我们对“不在模块级做 I/O”有相当严格的开发者约定,通常用简单的 Lazy 包装器来处理模块级对象。很好奇其他人在 Python 里用了什么办法。
ledauphin查看英文原文
I love the idea, but it feels like just an idea at this point. I'd rather read about them releasing their 'compile-time' analyzer and revealing their measurements for how much startup time it saves. In our codebase, we have pretty strict developer-enforced rules about not doing I/O at the module level, usually through the use of simple "Lazy" wrappers for module-level objects. I'd be curious to know what other approaches people have taken with Python here.
认为同等投入下拆微服务可能比严格模块检查带来更高的整体可靠性
有意思的是他们选择了这条路而不是微服务。这看起来很适合逐步拆分成微服务。有人因为非确定性结果的一次改动搞挂了生产环境?那不如把他们在改的那部分拆成独立服务。有了合适的降级方案,拆出去的那部分出问题就不至于再次拖垮整个生产环境。澄清一下:我不是说微服务能解决所有问题或工作量更小,只是说投入同等精力,拆成多个服务可能带来更高的整体可靠性,还能按任务选用不同语言、更低风险地更频繁部署,并把这类“导入即执行”的行为隔离到小得多的范围内。
jedberg查看英文原文
It's interesting to me that they are going down this path instead of the microservices path. This seems like something ripe for slowly breaking down into microservices. Someone made a change that took down production because of non-deterministic outcomes? How about break out whatever they were changing into it's own service? With proper fallbacks, breaking that part shouldn't take down all of production again. To be clear, I'm not saying microservices will solve all their problems or be less work. I'm just saying that with an equal level of effort, they would probably get more overall reliability by having multiple services, they'd be able to use multiple languages, whatever is suited to the task at hand, be able to deploy even more often with less risk, and be able to isolate these types of "change on import" behavior to a much smaller surface on any given deployment.
历次讨论
同类还值得读
开源应用架构
The Architecture of Open Source Applications
《开源软件架构》(AOSA)是一本在线免费书,邀请知名开源项目的作者亲自剖析其内部设计,涵盖 SQLAlchemy、LLVM、Audacity、Apache、Bash 等系统。
编程与软件工程上首页 9 次 · 2011–2024aosabook.org编辑精选编程语言简史:简短、不完整且多半错误
A Brief, Incomplete, and Mostly Wrong History of Programming Languages
一篇以戏谑、故意不准确的方式回顾编程语言史的博客文章,用夸张轶事和讽刺定义串起 Lisp、C、C++、Java、JavaScript、PHP 等语言。
编程与软件工程上首页 8 次 · 2009–2021james-iry.blogspot.comSQLite 是如何测试的
How SQLite is tested
SQLite 官方文档页,系统介绍 SQLite 的测试体系:多种测试方法(TCL 测试、模糊测试、回归测试、静态分析等)和海量测试用例,展示了这个小型开源项目如何靠极致测试获得…
编程与软件工程上首页 8 次 · 2009–2025sqlite.org编辑精选高级编程语言(2009)
Advanced programming languages (2009)
Matt Might 的经典文章,按用途推荐“高级编程语言”:如果追求就业就选主流语言,如果追求表达力、学术研究或创业,则推荐 Haskell、OCaml、Scala 等函数式语…
编程与软件工程上首页 8 次 · 2009–2022matt.might.net我的 Forth 与栈机器经历
Yosefk - My history with Forth & stack machines
作者以亲身经历回顾了使用 Forth 语言和栈式机器的历程:既赞叹其简洁优雅、代码量小、可自举的优点,也坦率指出栈操作在实际编程中的痛苦,以及 Forth 社区文化中的问题。
编程与软件工程上首页 8 次 · 2010–2024yosefk.com编辑精选