User-Agent: Cyrus-JMAP/3.3.0-248-gcd102cb-fm-20200901.001-gcd102cb9
Mime-Version: 1.0
Message-Id: <9e513706-7ebc-4a40-8e5e-a5f94d84bc77@www.fastmail.com>
In-Reply-To: <87ft81zvi0.fsf@euandre.org>
References: <d78064d2-5441-4275-96d9-463a08dfd7a9@www.fastmail.com>
 <87ft81zvi0.fsf@euandre.org>
Date: Tue, 01 Sep 2020 21:24:55 +1000
From: "Seph Gentle" <me@josephg.com>
To: EuAndreh <eu@euandre.org>, ~euandreh/public-inbox@lists.sr.ht
Subject: Re: Making a database
Content-Type: text/plain;charset=utf-8
Content-Transfer-Encoding: quoted-printable

On Tue, Sep 1, 2020, at 7:39 PM, EuAndreh wrote:
> "Seph Gentle" <me@josephg.com> writes:
>=20
> > I've been thinking along similar lines, and am talking to some folks=

> > about building something on top of CRDTs. Modern CRDTs like automerg=
e
> > and y.js are just getting to the point where they're fast and small
> > enough to work fine. I can see something really valuable that looks
> > kind of like git, except where you can store arbitrary data. And
> > live-sync peers - so if I make changes you see those changes live on=

> > your computer too - without (necessarily) needing a centralized
> > server.
> >
> > Love to chat if you & others are keen - I/we are still in the proces=
s
> > of figuring out what this thing will look like. But, I feel like the=

> > database piece is really core here too, and I like a lot of your
> > thinking in this space.
>=20
> Thanks for the encouragement :)
>=20
> I've looked at CRDTs, but I haven't done a deep dive yet, because my
> first impression was that they generally choose to limit the types of
> data structures and operations that you can perform on them in favour =
of
> being conflict-free.
>=20
> That's why I didn't mention it on my article directly, but I admit to
> having it as pre-requisite to making an informed decision on the
> subject.
>=20
> WDYT?

Well, they=E2=80=99re conflict-free in the sense that (unlike Postgres/f=
oundationdb/etc) the database doesn=E2=80=99t reject conflicting concurr=
ent updates. Which is a property you need if you=E2=80=99re going to com=
mit local writes while you=E2=80=99re offline. And if I=E2=80=99m readin=
g your blog post right it sounds like that=E2=80=99s what you=E2=80=99re=
 going for.

You kinda gave a good description of them in your post already. They wor=
k by requiring all peers to resolve concurrent edits to an object the sa=
me way. That could be last-writer-wins (based on time stamps like you sa=
id in your article). Or it could store all conflicting versions together=
 to be resolved by the next reader (Eg riak). Or it could do something m=
ore clever (like merge changes using automerge or y.js or the like). Git=
 is sort of a CRDT - it=E2=80=99s just it=E2=80=99s 3 way merge algorith=
m is a bit of a dogs breakfast as far as these things go.

Anyway, happy to chat about this stuff in more detail if you wanna zoom =
or something. Or hop into the automerge slack (with pvh from ink&switch,=
 and Martin Kleppmann, and others) and we can talk about this stuff - ht=
tps://join.slack.com/t/automerge/shared_invite/zt-e4p3760n-kKh7r3KRH1Yww=
NfiZM8ktw

As I said - I=E2=80=99m looking to build something like this too and it=E2=
=80=99d be good to compare notes on our approaches!