系统与基础设施github.com
rqlite:基于 SQLite 的轻量级分布式关系数据库
Rqlite: The lightweight, distributed relational database built on SQLite
2016 — 2022,它一次次回到首页
黑点是上首页(≥ 50 分)的那次投稿,空心点是没冲上去的投稿,点越大分数越高。
导读
rqlite 是一个用 Go 编写的轻量级分布式关系数据库,在 SQLite 之上叠加 Raft 共识层,使多个 SQLite 实例组成集群,每次变更要么复制到法定人数节点,要么完全不生效。该项目多年来反复登上 Hacker News 首页,讨论集中在复制粒度、一致性模型以及把本地数据库网络化后的代价。
为什么经久不衰:它把 SQLite 的极简与 Raft 的容错结合起来,让小型团队也能运行分布式 SQL;围绕它展开的设计权衡讨论历经多年仍然经典。
HN 评论精选
复制高层 SQL 变更在跨版本滚动升级时可能导致副本数据分叉,应改为复制底层块变更。
看起来这是把 Raft 叠加在 SQLite 之上。我不喜欢系统复制 'update users set salary = salary + 1000 where ...' 这类高层变更,更希望复制低层变更,比如“把键/块 X(内容应为 C_x)替换为 C_x'”。为什么?想象你在做滚动升级,一部分副本跑新版 SQLite、一部分跑旧版,它们执行高层查询的方式可能不完全一致。例如没有 order by 时 select 结果顺序是未定义的;假设有人做了依赖这一点的变更(如 insert ... select ... limit),数据库就会开始分叉——不仅在底层字节和实现相关的顺序上,还在真实行数据上。我参与过一个大型分布式系统,最初复制高层变更,后来因此改为复制低层变更;我们有检测副本不一致的机制,复制高层变更是差异的最大来源(硬件错误排第二,为此加了很多校验和)。
scottlamb查看英文原文
It looks like this layers Raft on top of SQLite. I don't like when systems replicate high-level changes like "update users set salary = salary + 1000 where ...;" Instead, I prefer they replicate low-level changes like "replace key/block X, which should have contents C_x, with C_x'". Why? Imagine you're doing a rolling update. Some of your replica are running the newer version of SQLite and some are running the older version. They may not execute the high-level query in exactly the same way. For example, in the absence of an "order by" clause, select results' order is unstated. So imagine someone makes a mutation that depends on this: "insert ... select ... limit". (Maybe a dumb example but it can happen anyway.) Now the databases start to diverge, not only in underlying bytes and implementation-defined ordering but in actual row data. I worked on a major distributed system that originally replicated high-level changes and switched to replicating low-level changes for this reason. We had a system for detecting when replicas didn't match, and replication of high-level changes was the biggest reason for diffs. (Hardware error was the second biggest reason; we added a lot of checksumming because of that.)
按抽象层次划分了三种 SQLite 分布式化方案。
看起来现在可以在不同层面把 SQLite 分布式化:语句层(rqlite)、VFS 层(canonical/dqlite)、块层(benbjohnson/litestream)。对这个优秀项目的很酷的增强!
mech422查看英文原文
So it looks like you can now distribute SQLite at the: Stmt level: https://github.com/rqlite/rqlite VFS Level: https://github.com/canonical/dqlite Block Level: https://github.com/benbjohnson/litestream Really cool enhancements to an awesome project!
提出疑问:并发的两个变更若写到不同多数派,如何保证一致性。
我对共识算法和分布式系统一无所知,请包涵。rqlite 用 Raft 达成共识,我理解是三个实例时每个变更至少写到两个。但如果两个变更分别写到不同的一对实例呢?那三个实例就有三个不同版本的数据:比如变更 X 写到 A 和 B,变更 Y 写到 B 和 C,现在 A 有 X,B 有 X 和 Y,C 只有 Y,怎么决定谁对?
blackbear_查看英文原文
I know nothing of consensus algorithms and distributed systems so bear with me please. > rqlite uses Raft to achieve consensus across all the instances of the SQLite databases, ensuring that every change made to the system is made to a quorum of SQLite databases, or none at all. What I understood from this sentence is that, if we have three instances, rqlite will make sure that every change is written to at least two. But what if two changes are written to two different pairs of instances? Then the three instances will have three different versions of the data. For example, change X is written to instances A and B, and change Y is written to B and C. Now A has X, B has X and Y, and C has Y only. How do you decide who is right?
数据库移到网络后,原本可接受的 n+1 查询模式会变成性能瓶颈。
从本地数据库变成网络数据库至少有一个风险:SQLite 文档允许开发者写 n+1 风格查询,当数据库与应用在同一文件系统时这种模式没问题;但一旦加上网络调用,它就可能变成瓶颈。
mshenfield查看英文原文
Going from a local db to one over a network has at least one risk. The SQLite docs gives developers the okay to write "n+1" style queries (https://www.sqlite.org/np1queryprob.html). When the db is on the same file system as the application this pattern is fine. But as soon a you add a network call it becomes a potential bottleneck.
历次讨论
同类还值得读
TTY 揭秘(2008)
The TTY demystified (2008)
经典文章深入剖析了 Unix/Linux 中 TTY 子系统的工作原理,包括线路规程、终端驱动、信号与控制字符等机制。
系统与基础设施上首页 11 次 · 2009–2025linusakesson.net编辑精选为什么 GNU grep 这么快
Why GNU grep is Fast
FreeBSD 邮件列表上一篇广为流传的帖子,解释 GNU grep 为什么快:核心在于 Boyer-Moore 字符串搜索算法配合精心优化的内部循环和 I/O,让程序在搜索时“…
系统与基础设施上首页 9 次 · 2010–2022lists.freebsd.orgXplain:为普通人讲解 X11
Xplain – Explaining X11 for the rest of us
《Xplain》是作者(HN 用户 Jasper_)写的一系列 X11 长文,用他在浏览器中重建的 X11 绘图模型做交互演示,讲解 X11 的绘图模型与窗口系统设计。
系统与基础设施上首页 6 次 · 2013–2020magcius.github.io编辑精选旋风教程:为 Linux 创建极小 ELF 可执行文件(1999)
A Whirlwind Tutorial on Creating Really Teensy ELF Executables for Linux (1999)
breadbox 写于 1999 年的经典教程,主题是如何为 Linux 手工构造并裁剪出体积极小的 ELF 可执行文件。
系统与基础设施上首页 6 次 · 2007–2025muppetlabs.comPostgreSQL 内部原理
The internals of PostgreSQL
《The Internals of PostgreSQL》是一本免费在线书籍,系统讲解 PostgreSQL 的内部实现,包括查询处理、并发控制、WAL、缓冲区管理等核心机制。
系统与基础设施上首页 5 次 · 2017–2025interdb.jp编辑精选