互联网与文化annas-blog.org编辑精选
如何成为一名海盗档案管理员
How to become a pirate archivist
2022 — 2024，它一次次回到首页
黑点是上首页（≥ 50 分）的那次投稿，空心点是没冲上去的投稿，点越大分数越高。
导读
这是 Anna's Blog 上关于“如何成为盗版档案管理员”的文章，讨论数字资料的采集、保存以及通过种子等方式分发。评论围绕 Sci-Hub、What.cd/Redacted 等不同档案模式，以及抓取工具和长期做种展开。
为什么经久不衰：它把“保存知识”的理想落到可操作的技术与分发策略上，并引出至今仍相关的版权、匿名与可持续归档问题。
HN 评论精选
比较 Sci-Hub 的公开分发与私有 tracker 分享率经济的可持续性和可复制性。
我很好奇 Sci-Hub 的方式与 What.cd/Redacted 的方式相比如何。据我理解，Sci-Hub 靠足够好的用户体验收集科学文档，借助科研人员的好感让他们上传文档，并采用公开的闪电战式分发机制；如果硬盘够大、网速够快，现在就可以开始下载整个库，看能否跑赢版权方。另一方面，What.cd/Redacted 似乎用 BitTorrent 分享率建立私有 tracker 经济：新用户加入时有一定免费下载量，但由于 1:1 上传/下载比和少数先行者坐拥巨大分享率，其他人会争相上传新的 FLAC 来提升分享率以继续下载。这意味着整个库无法随意轻松复制。然而该 tracker 作为 What.cd 已被关停，后来以 Redacted 重现，两个服务之间有没有数据丢失？另外 rutracker 似乎是另一种方式：闪电战式添加并发布内容，代价似乎是内容质量。
jancsika查看英文原文
I'm curious how Sci-hub's approach compares to the What.cd/Redacted approach. IIUC Sci-hub has scooped up science docs through a good enough UX that it was able to leverage the goodwill of science folks to upload docs (plus whatever other methods it has used to scoop up docs), and it uses a public blitzkrieg-style distribution mechanism. I.e., I guess if one had a big enough harddrive and a fast enough internet connection, one could start downloading the lib right now and see if they win the race against the copyright holders. On the other hand, the What.cd/Redacted approach seems to use Bittorrent ratios to create a private-tracker economy. New users get a few gigs free download on joining. But apparently because a) there's a 1:1 upload/download ratio, and b) a few first-mover fat cats are sitting on enormous ratios, this means there is a scramble by everyone else to upload new FLACs to build up their ratio so they can continue to be able to download FLACs. It seems that would mean the library-in-its-entirety cannot be easily replicated at will. Yet the tracker was apparently already nuked off the internet as What.cd and reappeared later as Redacted. Was any data lost between the two services? Oh yeah, there's also apparently another approach in rutracker, which seems to be blitzkrieg to add content and publish, at the (apparent?) cost of quality of content. It's really a shame that the nerdy, completist domain of digital archiving through torrents isn't covered by fair use. Perhaps we could exclude the most recent 10 years of music so that the hopeful young musician streamers can get paid a few hundred dollars for millions of streams and then receive the silver lining of fair use protection against a label refusing to release one of their albums.
Sci-Hub 创始人因服务器错误信息泄露路径和用户名而被识别。
我认为 Alexandra Elbakyan 其实并不想被揭露为 Sci-Hub 背后的馆员，是她糟糕的 opsec 导致身份暴露。基本上，她的服务器被设置成输出 PHP 详细错误信息，包括出错源文件的完整路径，路径在 /home/ringo-ring 目录下，而这个用户名可追溯到她在一个无关网站上的用户名，并关联到真名。在曝光之前她是匿名的。
ynno查看英文原文
I think Alexandra Elbakyan actually did not want to be revealed as the librarian behind Sci-Hub, it was her poor opsec that led to her being identified. Basically her servers were set up to emit detailed error messages from PHP, including full path of faulting source file, which was under directory /home/ringo-ring, which could be traced to a username she had online on an unrelated site, attached to her real name. Before this revelation, she was anonymous.
现有种子分发对文件移动和部分保存不友好，需要按文件发现、可持久做种的 p2p 系统。
作为一个活跃的囤积者，我认为“6. 分发：打包成种子、在某个地方发布、让人们传播”有问题。我怀念一种 p2p 应用：带有种子打包和类似 Kademlia 的按文件广告与发现，我可以把它指向 NAS 上的一堆随机文件目录，它们能跟已发布的种子对接。这样即使种子只完成一部分，也能活得更久。额外选项是，当有人请求我索引并广告过的文件时，应用能通知我加载 DVD。多年来我的程序偏好变了、文件位置变了——我移动文件、让它们离线、刻到 DVD、删掉种子的一部分只为留下有趣内容。现在这些种子丢了，至少我的做种贡献没了。但我几乎从不改这些文件的内容，校验和永远不变，所以它们仍应可被发现。数字保存需要更好的分发系统。
imhoguy查看英文原文
As an active hoarder I think there is problem with "6. Distribution: Packaging it up in torrents, announcing it somewhere, getting people to spread it.". I miss a p2p application with torrent packaging and Kademlia like per-file advertising and discovery, where I could point it to my hefty NAS directory of random things and they could be wired to released torrents. This way we could make torrents live much longer, even partially complete. In super extra option the app could even notify me to load DVD because somebody asked for a file which I indexed and advertised previously. For years my program preferences changed, file locations changed - I have moved files around, made them offline, burned on DVDs, deleted some parts of torrent just to keep interesting stuff. Now these torrents are lost, at least my seeding contribution is gone. But I almost never change the content of these files, their checksum stays the same forever, so they could be still discoverable. The digital preservation needs better distribution system.
推荐用 ZAP+Firefox 抓取浏览内容、用 Scrapy 跟随分页并管理下载与去重。
我不在盗版档案管理领域，但第 3 和第 5 节与我的兴趣相关。我用 ZAP 配合一份 Firefox 来归档浏览站点时看到的所有内容，效果很好，因为 Firefox 允许调整浏览器代理而不用改系统代理。它甚至实现了所述目标：把 HTML 等存进数据库，因为 ZAP 用 hsqldb。第 5 节读起来像 Scrapy 的广告，因为它非常擅长跟随所有分页链接，然后把提取内容作为数据结构输出，和/或告诉 Scrapy 要原样下载某些媒体。默认情况下，它会把本地内容放进你选的目录，并对 URL 做哈希作为本地文件名；一个单独的 json 文件充当下载内容与哈希后磁盘文件名之间的“账本”。Scrapy 还能把第 3 和第 5 节粘合起来，因为它有可插拔的重复检测钩子和 HTTP 缓存支持。
mdaniel查看英文原文
I'm not in the pirate archivist space, but sections 3 and 5 are relevant to my interests. I've had great luck with ZAP (https://github.com/zaproxy/zaproxy#readme) glued to a copy of Firefox (because it allows monkeying with the _browser_'s proxy without having to alter the system one as other browsers do) for archiving all content seen while surfing around a site. It even achieves the stated goal of preserving the HTML (etc) in a database since ZAP uses hsqldb Then, section 5 reads like an advertisement for Scrapy since it is just stellar at following all pagination links and then either emitting the extracted payload as your own data structure and/or by telling Scrapy you want to download some media as-is. It will, by default, put the local content in a directory of your choice and hash the url to make the local filename. A separate json file serves as the "accounting" between the things it downloaded and their hashed on-disk filename Scrapy is also able to glue 3 and 5 together because it has a pluggable (everything, heh) dupe detection hook and also HTTP cache support that can be backed by anything, including the aforementioned hsqldb operating in network mode. Scrapy is also very test friendly, since each method accepts a well known python object and emits either a follow-on request, zero or more extracted objects, or nothing if pagination has ended. Thus, one need not rerun the whole scraping process just to test if a bug has been fixed, or during development I can appreciate there may be other scraping frameworks, but of the ones I've tried Scrapy makes everything that I've asked it to do simple and transparent
历次讨论
同类还值得读
如今电脑上几乎所有操作在感知上都比 1983 年更慢（2017）
Almost everything on computers is perceptually slower than it was in 1983 (2017)
2017 年的推文串认为，尽管计算机硬件比 1983 年快得多，日常操作的感知速度却常常更慢：启动、界面响应和网络应用被层层抽象、动画、后台检查与网络依赖拖累。
互联网与文化上首页 5 次 · 2017–2020twitter.com网站肥胖危机
The Website Obesity Crisis
Maciej Cegłowski（idlewords）的经典演讲文稿《网站肥胖症危机》，批评现代网页越来越臃肿：动辄数 MB 的 JavaScript、追踪脚本和广告，把本可以轻…
互联网与文化上首页 5 次 · 2016–2021idlewords.com编辑精选亚马逊那本标价 23,698,655.93 美元的苍蝇书
Amazon’s $23,698,655.93 book about flies
博文讲的是亚马逊上一本关于果蝇的二手书被标到 23,698,655.93 美元。
互联网与文化上首页 5 次 · 2011–2022michaeleisen.org浏览器 User-Agent 字符串的历史
History of the browser user-agent string (2008)
2008 年的经典文章用戏谑的“创世纪”口吻，讲述浏览器 User-Agent 字符串如何从简单的标识演变成层层叠加的伪装：浏览器为了绕过按 UA 分发的网站检测，纷纷把自己伪装…
互联网与文化上首页 5 次 · 2012–2022webaim.org编辑精选最后的问题——艾萨克·阿西莫夫
The Last Question - Isaac Asimov
阿西莫夫最著名的短篇科幻小说《最后的问题》的在线全文。
互联网与文化上首页 5 次 · 2009–2019multivax.com编辑精选