Free Ekanayaka
2017-08-19 08:46:59 UTC
Hi,
first of all let me thank Richard Hipp and the rest of the SQLite team
and community for such a great piece of software. I've been recently
working with SQLite's code base and found it an absolute pleasure to
read (and have learned from it too).
In this mail I'd like to:
1) Present dqlite, a library replicating your application's SQLite
database across N nodes and safely surviving any minority of them
dying or disconnecting (only for Go applications for now, see below).
2) Submit a patch to SQLite that introduces a minimal replication API,
and get feedback about its possible inclusion upstream.
= dqlite =
It's a Go package that uses the Raft algorithm to replicate SQLite WAL
frames across a cluster of nodes. This roughly means that you can open a
SQLite connection using the "database/sql" standard lib API and have
anything you transactionally replicated. No external process needed.
Ideally this library should have been written in C or Rust, to support
binding to any language. However, due to the use case and timeline of
the first project that will use it (LXD [0]), and due to the lack of
mature Raft implementations in C/Rust, Go was chosen instead. It should
hopefully at least serve as reference to anyone needing a C/Rust
version.
The work has been funded by Canonical, the company behind Ubuntu. Please
see the dqlite's home page [1] for more details.
= SQLite replication API patch =
This is the SQLite patch that dqlite depends on. It essentially adds a
few key hooks in the pager and write-ahead log to let external libraries
like dqlite implement WAL-based database replication.
As you'll quickly see, it's by no means ready for upstream inclusion, in
particular it lacks unit tests and more comprehensive documentation
comments (note however that virtually every code path introduced by the
patch is already exercised indirectly by dqlite's own unit tests).
If the SQLite team thinks there is room for upstream inclusion, I'll be
more than glad to do the necessary work to make the patch adhere to
SQLite's standards and go through a review process.
The patch has currently 703 additions and 22 deletions, and is published
on GitHub [2].
Cheers,
Free
[0] https://linuxcontainers.org/
[1] https://github.com/CanonicalLtd/dqlite
[2] https://github.com/CanonicalLtd/sqlite/commit/2a9aa8b056f37ae05f38835182a2856ffc95aee4
first of all let me thank Richard Hipp and the rest of the SQLite team
and community for such a great piece of software. I've been recently
working with SQLite's code base and found it an absolute pleasure to
read (and have learned from it too).
In this mail I'd like to:
1) Present dqlite, a library replicating your application's SQLite
database across N nodes and safely surviving any minority of them
dying or disconnecting (only for Go applications for now, see below).
2) Submit a patch to SQLite that introduces a minimal replication API,
and get feedback about its possible inclusion upstream.
= dqlite =
It's a Go package that uses the Raft algorithm to replicate SQLite WAL
frames across a cluster of nodes. This roughly means that you can open a
SQLite connection using the "database/sql" standard lib API and have
anything you transactionally replicated. No external process needed.
Ideally this library should have been written in C or Rust, to support
binding to any language. However, due to the use case and timeline of
the first project that will use it (LXD [0]), and due to the lack of
mature Raft implementations in C/Rust, Go was chosen instead. It should
hopefully at least serve as reference to anyone needing a C/Rust
version.
The work has been funded by Canonical, the company behind Ubuntu. Please
see the dqlite's home page [1] for more details.
= SQLite replication API patch =
This is the SQLite patch that dqlite depends on. It essentially adds a
few key hooks in the pager and write-ahead log to let external libraries
like dqlite implement WAL-based database replication.
As you'll quickly see, it's by no means ready for upstream inclusion, in
particular it lacks unit tests and more comprehensive documentation
comments (note however that virtually every code path introduced by the
patch is already exercised indirectly by dqlite's own unit tests).
If the SQLite team thinks there is room for upstream inclusion, I'll be
more than glad to do the necessary work to make the patch adhere to
SQLite's standards and go through a review process.
The patch has currently 703 additions and 22 deletions, and is published
on GitHub [2].
Cheers,
Free
[0] https://linuxcontainers.org/
[1] https://github.com/CanonicalLtd/dqlite
[2] https://github.com/CanonicalLtd/sqlite/commit/2a9aa8b056f37ae05f38835182a2856ffc95aee4