编程与软件工程scrapingbee.com
Python 网页抓取入门
Web Scraping 101 with Python
2021 — 2022，它一次次回到首页
黑点是上首页（≥ 50 分）的那次投稿，空心点是没冲上去的投稿，点越大分数越高。
导读
这篇 ScrapingBee 的入门文章介绍用 Python 做网页抓取的基本流程和常见工具，适合新手了解从发起请求到提取数据的套路。评论者补充了 pandas.read_html、wget 镜像、Selenium/Playwright 等更实用的做法，也争论了静态解析与无头浏览器的取舍。
为什么经久不衰：它提供了抓取入门的路线图，而评论区沉淀了多年实战技巧和架构反思，今天读仍有参考价值。
HN 评论精选
HTML 表格可直接用 pandas.read_html 一行提取。
在跳进各种框架之前，如果你的数据恰好存放在 HTML 表格里：import pandas as pd; dfs = pd.read_html(url)。其中 dfs 是 dataframe 数组，页面每个 HTML 表格对应一项。
philshem查看英文原文
Before jumping into frameworks, if your data is lucky enough to be stored in an html table: import pandas as pd dfs = pd.read_html(url) Where ‘dfs’ is an array of dataframes - one item for each html table on the page. https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_html.html
先用 wget 镜像到本地，迭代更快且降低对站点影响。
分享一个技巧：抓网站数据时，先用 wget 镜像模式把有用页面下载到本地。本地迭代解析快得多，也更不容易误伤网站或引起主机注意。
VBprogrammer查看英文原文
One tip I would pass on when trying to scrape data from a website, start by using wget in mirror mode to download the useful pages. It's much faster to iterate on scraping the data once you have it locally. Also, less likely to accidentally kill the site or attract the attention of the host.
实战中 JS 链接最棘手，Selenium 方案胜出且需去重。
我曾为律师团队做网站镜像，试过 httrack、无头 Chromium/Selenium 等多种方案。最大的麻烦是处理 JavaScript 链接；最后 Python + Selenium 成了胜出组合，而它是我最后才尝试的。性能上很关键的一点是不要重复访问同一页面，因为页眉页脚里的链接会重复成百上千次。
cubano查看英文原文
My last contract job was to build a 100% perfect website mirroring program for a group of lawyers who were interested in building class action lawsuits against some of the more henious scammers out there. I ended up building like 8 versions of it, literally using every PHP and Python library and resource I could find. I tried httrack, php-ultimate-web-scraper (from github), headless chromium. headless selenium, and a few others By far the biggest problem was dealing with JS links...you wouldn't think from the start it would be such a big deal but yet..it was. Selenium with python turned out to be the winning combination, and of course, it was the last one I tried. Also, this is an ideal project to implement recursion altho you have to be careful about exit conditions. One thing that was VERY important for performance was not visiting any page more then once because, obviously, certain links in headers and footers are duped sometimes 100s of times. JS links often made it very difficult to discover the linked page, are certain library calls that were supposed to get this info for you often didn't work. It was a super fun project, and in the end considering I only worked for 2 months, I shipped some decent code that was getting like 98.6% of the pages perfectly. The final presentation was interesting...for some reason my client I think got in his head that I wasn't very good programmer or something, and as we ran thru his list of sample sites expecting my program to error out or incorrectly mirror the site, but it handled all 10 of the sites about perfectly and he was rather flabbergasted because he told me it would have taken him a week hand clicking the site for the mirror but instead the program did them all in under an hour.
应优先无头浏览器，推荐 Playwright 而非先讲 urllib3/Selenium。
我认为这篇文章的整体软件架构思路根本是反的。如今大量网页是客户端渲染，应该一开始就选无头方案：启动真实浏览器并自动化操作，和真人使用浏览器无异。若对重度客户端渲染的页面用 urllib3，服务器返回的 HTML 是不完整的。只有较少见的情况下——静态 HTML——才用 urllib3。文章提到 Selenium，但我听说它不是最好；用 Python 的话 Playwright 似乎是最佳选择，由前 Chrome DevTools 团队打造。
kaycebasques查看英文原文
I think the overall software architecture approach of this post is fundamentally backwards. Given how much of the web is rendered client-side these days you need to start out with a headless option. Headless means that you fire up a true browser and then automate the actions that you need to perform. It's indistinguishable from a real person using a browser. If you try to use urllib3 on a webpage that does heavy client-side rendering then you're going to get incomplete HTML returned from the server (because the website intends to use JavaScript to complete the rendering of the page). On the more rare occasions when you are dealing with static HTML (i.e. there is no rendering on the client; the HTML returned from the server is the complete content) then you can use something like urllib3. Re: which headless library to use this post mentions Selenium which was one of the first headless libs but from what I've heard probably not the best (in terms of developer experience or reliability or robustness) but that's only hearsay... I've never used Selenium myself. Playwright seems like the best option in town if you want to use Python. Built by the former Chrome DevTools team (meaning those people really know how browser internals work). https://playwright.dev/python/docs/intro
历次讨论
同类还值得读
开源应用架构
The Architecture of Open Source Applications
《开源软件架构》（AOSA）是一本在线免费书，邀请知名开源项目的作者亲自剖析其内部设计，涵盖 SQLAlchemy、LLVM、Audacity、Apache、Bash 等系统。
编程与软件工程上首页 9 次 · 2011–2024aosabook.org编辑精选编程语言简史：简短、不完整且多半错误
A Brief, Incomplete, and Mostly Wrong History of Programming Languages
一篇以戏谑、故意不准确的方式回顾编程语言史的博客文章，用夸张轶事和讽刺定义串起 Lisp、C、C++、Java、JavaScript、PHP 等语言。
编程与软件工程上首页 8 次 · 2009–2021james-iry.blogspot.comSQLite 是如何测试的
How SQLite is tested
SQLite 官方文档页，系统介绍 SQLite 的测试体系：多种测试方法（TCL 测试、模糊测试、回归测试、静态分析等）和海量测试用例，展示了这个小型开源项目如何靠极致测试获得…
编程与软件工程上首页 8 次 · 2009–2025sqlite.org编辑精选高级编程语言（2009）
Advanced programming languages (2009)
Matt Might 的经典文章，按用途推荐“高级编程语言”：如果追求就业就选主流语言，如果追求表达力、学术研究或创业，则推荐 Haskell、OCaml、Scala 等函数式语…
编程与软件工程上首页 8 次 · 2009–2022matt.might.net我的 Forth 与栈机器经历
Yosefk - My history with Forth & stack machines
作者以亲身经历回顾了使用 Forth 语言和栈式机器的历程：既赞叹其简洁优雅、代码量小、可自举的优点，也坦率指出栈操作在实际编程中的痛苦，以及 Forth 社区文化中的问题。
编程与软件工程上首页 8 次 · 2010–2024yosefk.com编辑精选