编程与软件工程incolumitas.com
像大厂一样抓取网页
Scrape like the big boys
2021 — 2024，它一次次回到首页
黑点是上首页（≥ 50 分）的那次投稿，空心点是没冲上去的投稿，点越大分数越高。
导读
这篇文章讨论大规模网页抓取与反爬之间的攻防，涉及代理 IP、移动网络、登录与限速、行为检测等为什么难以彻底封堵。文中强调抓取方和防守方都在持续升级手段，最终更像一场长期的军备竞赛。
为什么经久不衰：它把反爬与抓取还原成真实的工程攻防和成本取舍，多年后对做数据采集或站点防护的人仍有参考价值。
HN 评论精选
一线反爬经验：封 IP 和强制登录都难奏效，按登录限速、限制数据范围和数据投毒更实用，但本质是军备竞赛。
我曾在一家富时100公司负责系统工程师团队。我们的数据很有价值，但只在短时间内有价值。我们一直被爬，带来托管等成本。甚至看到竞争对手使用我们的数字（守规矩的用来调整自己的价格，不守规矩的直接照搬）。正如文章所说，我们无法封掉移动运营商 IP，有些 IP 后面有超过 10 万用户。强制用户登录效果有限，因为爬虫直接注册账号。我们用了几个办法减少抓取：按登录限速、把数据限制在已知工作流……但最有效的是取消限制并开始给“坏”数据。所谓坏，是把价格小幅上调或下调。这让他们蒙受损失，但也不是银弹。如果客户按被改过的数字做了交易，我们会告知并按正确价格成交。这是个很酷的问题，但终究是军备竞赛。
biosed查看英文原文
I used to lead Sys Eng for a FTSE 100 company. Our data was valuable but only for a short amount of time. We were constantly scraped which cost us in hosting etc. We even seen competitors use our figures (good ones used it to offset their prices, bad ones just used it straight). As the article suggest, we couldn't block mobile operator IPs, some had over 100k customers behind them. Forcing the users to login did little as the scrapers just created accounts. We had a few approaches that minimised the scraping: Rate Limiting by login, Limiting data to know workflows ... But our most fruitful effort was when we removed limits and started giving "bad" data. By bad I mean alter the price up or down by a small percentage. This hit them in the pocket but again, wasn't a golden bullet. If the customer made a transaction on the altered figure we we informed them and took it at the correct price. It's a cool problem to tackle but it is just an arms race.
用移动代理抓无 API 的政府网站；cgNAT 使封 IP 会误伤大量普通用户，几乎不可行。
我经常爬政府网站，因为它们不提供 API。移动代理方面，我用 proxidize 的 dongle 和 mobinet.io（免费，用 Android 设备）。正如文章所说，有了 cgNAT，基本不可能封它们；在我的例子里，如果放在几个地点、每个地点用一个运营商，一封就会让半个国家无法访问网站。
abc03查看英文原文
I scrap government sites a lot as they don't provide apis. For mobile proxies, I use the proxidize dongles and mobinet.io (free, with Android devices). As stated in the article, with cgNAT it's basically impossible to block them as in my case, half the country couldn't access the sites anymore (if you place them in several locations and use one carrier each there).
用本地 Firefox 加脚本驱动浏览器处理难爬站点，适合小规模数据录入，不可扩展但便于观察调试。
在一个特别难爬、带某种我始终无法稳定搞定的 bot 防护的网站上（如果有人想知道具体是什么，我可以去查），我现在用一台小 Intel NUC 跑 Firefox，监听本地服务器，并用 Temper Monkey 执行命令。效果很好，我能实际看到它在做什么、哪里出错。（当然不可扩展。）我们用它在一个政府网站做数据录入。人类平均要点点打打 10 分钟，机器人大概 10 秒。去年我们做了 12000 条。好机器人。
neals查看英文原文
In a particularly hard to scrape website, using some kind of bot protection that I just couldn't reliably get working (if anybody wants to know what that was exactly, I'll go and check it) I now have a small Intel NUC running with firefox that listens to a local server and uses Temper Monkey to perform commands. Works like a charm and I can actualy see what it's doing and where it's going wrong. (though it's not scalable, of course) We use it for data-entry on a government website. A human would average around 10 minutes of clicking and typing, where the bot takes maybe 10 seconds. Last year we did 12000 entries. Good bot.
与其封 IP 或指纹，不如分析点击行为；识别后返回静态假页面，降低对抗成本且更难被爬虫发现。
我工作过的地方不再关心 IP、浏览器等，因为那只是竞速。我们做的是分析点击行为并据此行动。识别出爬虫后，就给它返回假页面。因为假页面是静态的，稍微降低了成本。总体上，他们要花很长时间才发现模式，对我们来说可管理得多。
InvOfSmallC查看英文原文
Where I was working we stopped caring about ips browser etc because it was just a race. What we did was analyzing behaviour of clicks and acted on that. When we recognized it we went on serving a fake page. It cuts down a little bit of costs because it was static pages. In general it took a lot of time for them to discover the pattern and it was way more manageable for us.
历次讨论
同类还值得读
开源应用架构
The Architecture of Open Source Applications
《开源软件架构》（AOSA）是一本在线免费书，邀请知名开源项目的作者亲自剖析其内部设计，涵盖 SQLAlchemy、LLVM、Audacity、Apache、Bash 等系统。
编程与软件工程上首页 9 次 · 2011–2024aosabook.org编辑精选编程语言简史：简短、不完整且多半错误
A Brief, Incomplete, and Mostly Wrong History of Programming Languages
一篇以戏谑、故意不准确的方式回顾编程语言史的博客文章，用夸张轶事和讽刺定义串起 Lisp、C、C++、Java、JavaScript、PHP 等语言。
编程与软件工程上首页 8 次 · 2009–2021james-iry.blogspot.comSQLite 是如何测试的
How SQLite is tested
SQLite 官方文档页，系统介绍 SQLite 的测试体系：多种测试方法（TCL 测试、模糊测试、回归测试、静态分析等）和海量测试用例，展示了这个小型开源项目如何靠极致测试获得…
编程与软件工程上首页 8 次 · 2009–2025sqlite.org编辑精选高级编程语言（2009）
Advanced programming languages (2009)
Matt Might 的经典文章，按用途推荐“高级编程语言”：如果追求就业就选主流语言，如果追求表达力、学术研究或创业，则推荐 Haskell、OCaml、Scala 等函数式语…
编程与软件工程上首页 8 次 · 2009–2022matt.might.net我的 Forth 与栈机器经历
Yosefk - My history with Forth & stack machines
作者以亲身经历回顾了使用 Forth 语言和栈式机器的历程：既赞叹其简洁优雅、代码量小、可自举的优点，也坦率指出栈操作在实际编程中的痛苦，以及 Forth 社区文化中的问题。
编程与软件工程上首页 8 次 · 2010–2024yosefk.com编辑精选