~euandreh/public-inbox

5 2

Making a database

Seph Gentle
Details
Message ID
<d78064d2-5441-4275-96d9-463a08dfd7a9@www.fastmail.com>
DKIM signature
pass
Download raw message
I've been thinking along similar lines, and am talking to some folks about building something on top of CRDTs. Modern CRDTs like automerge and y.js are just getting to the point where they're fast and small enough to work fine. I can see something really valuable that looks kind of like git, except where you can store arbitrary data. And live-sync peers - so if I make changes you see those changes live on your computer too - without (necessarily) needing a centralized server.

Love to chat if you & others are keen - I/we are still in the process of figuring out what this thing will look like. But, I feel like the database piece is really core here too, and I like a lot of your thinking in this space.

-Seph
Details
Message ID
<87ft81zvi0.fsf@euandre.org>
In-Reply-To
<d78064d2-5441-4275-96d9-463a08dfd7a9@www.fastmail.com> (view parent)
DKIM signature
pass
Download raw message
"Seph Gentle" <me@josephg.com> writes:

> I've been thinking along similar lines, and am talking to some folks
> about building something on top of CRDTs. Modern CRDTs like automerge
> and y.js are just getting to the point where they're fast and small
> enough to work fine. I can see something really valuable that looks
> kind of like git, except where you can store arbitrary data. And
> live-sync peers - so if I make changes you see those changes live on
> your computer too - without (necessarily) needing a centralized
> server.
>
> Love to chat if you & others are keen - I/we are still in the process
> of figuring out what this thing will look like. But, I feel like the
> database piece is really core here too, and I like a lot of your
> thinking in this space.

Thanks for the encouragement :)

I've looked at CRDTs, but I haven't done a deep dive yet, because my
first impression was that they generally choose to limit the types of
data structures and operations that you can perform on them in favour of
being conflict-free.

That's why I didn't mention it on my article directly, but I admit to
having it as pre-requisite to making an informed decision on the
subject.

WDYT?
Seph Gentle
Details
Message ID
<9e513706-7ebc-4a40-8e5e-a5f94d84bc77@www.fastmail.com>
In-Reply-To
<87ft81zvi0.fsf@euandre.org> (view parent)
DKIM signature
pass
Download raw message
On Tue, Sep 1, 2020, at 7:39 PM, EuAndreh wrote:
> "Seph Gentle" <me@josephg.com> writes:
> 
> > I've been thinking along similar lines, and am talking to some folks
> > about building something on top of CRDTs. Modern CRDTs like automerge
> > and y.js are just getting to the point where they're fast and small
> > enough to work fine. I can see something really valuable that looks
> > kind of like git, except where you can store arbitrary data. And
> > live-sync peers - so if I make changes you see those changes live on
> > your computer too - without (necessarily) needing a centralized
> > server.
> >
> > Love to chat if you & others are keen - I/we are still in the process
> > of figuring out what this thing will look like. But, I feel like the
> > database piece is really core here too, and I like a lot of your
> > thinking in this space.
> 
> Thanks for the encouragement :)
> 
> I've looked at CRDTs, but I haven't done a deep dive yet, because my
> first impression was that they generally choose to limit the types of
> data structures and operations that you can perform on them in favour of
> being conflict-free.
> 
> That's why I didn't mention it on my article directly, but I admit to
> having it as pre-requisite to making an informed decision on the
> subject.
> 
> WDYT?

Well, they’re conflict-free in the sense that (unlike Postgres/foundationdb/etc) the database doesn’t reject conflicting concurrent updates. Which is a property you need if you’re going to commit local writes while you’re offline. And if I’m reading your blog post right it sounds like that’s what you’re going for.

You kinda gave a good description of them in your post already. They work by requiring all peers to resolve concurrent edits to an object the same way. That could be last-writer-wins (based on time stamps like you said in your article). Or it could store all conflicting versions together to be resolved by the next reader (Eg riak). Or it could do something more clever (like merge changes using automerge or y.js or the like). Git is sort of a CRDT - it’s just it’s 3 way merge algorithm is a bit of a dogs breakfast as far as these things go.

Anyway, happy to chat about this stuff in more detail if you wanna zoom or something. Or hop into the automerge slack (with pvh from ink&switch, and Martin Kleppmann, and others) and we can talk about this stuff - https://join.slack.com/t/automerge/shared_invite/zt-e4p3760n-kKh7r3KRH1YwwNfiZM8ktw

As I said - I’m looking to build something like this too and it’d be good to compare notes on our approaches!
Details
Message ID
<87pn75xtx4.fsf@euandre.org>
In-Reply-To
<9e513706-7ebc-4a40-8e5e-a5f94d84bc77@www.fastmail.com> (view parent)
DKIM signature
pass
Download raw message
> You kinda gave a good description of them in your post already. They
> work by requiring all peers to resolve concurrent edits to an object
> the same way. That could be last-writer-wins (based on time stamps
> like you said in your article). Or it could store all conflicting
> versions together to be resolved by the next reader (Eg riak). Or it
> could do something more clever (like merge changes using automerge or
> y.js or the like). Git is sort of a CRDT - it’s just it’s 3 way merge
> algorithm is a bit of a dogs breakfast as far as these things go. 

My ignorace is really on the applicability of it. The "store all
conflicting versions together or to be resolved by the next reader"
sounds like what I'm thinking about, so maybe I'm trying to re-invent
CRDTs after all.

I've just bumped the priority of this topic to better understand those
nuances.
Seph Gentle
Details
Message ID
<bab08df6-be8e-47bd-a254-7b86c41a9b47@www.fastmail.com>
In-Reply-To
<87pn75xtx4.fsf@euandre.org> (view parent)
DKIM signature
pass
Download raw message
On Wed, Sep 2, 2020, at 3:56 AM, EuAndreh wrote:
> > You kinda gave a good description of them in your post already. They
> > work by requiring all peers to resolve concurrent edits to an object
> > the same way. That could be last-writer-wins (based on time stamps
> > like you said in your article). Or it could store all conflicting
> > versions together to be resolved by the next reader (Eg riak). Or it
> > could do something more clever (like merge changes using automerge or
> > y.js or the like). Git is sort of a CRDT - it’s just it’s 3 way merge
> > algorithm is a bit of a dogs breakfast as far as these things go. 
> 
> My ignorace is really on the applicability of it. The "store all
> conflicting versions together or to be resolved by the next reader"
> sounds like what I'm thinking about, so maybe I'm trying to re-invent
> CRDTs after all.
> 
> I've just bumped the priority of this topic to better understand those
> nuances. 

This is worth watching if you haven’t seen it. Martin is at the forefront of the field - but this gives you a sense of where the research tech is at:

https://martin.kleppmann.com/2020/07/06/crdt-hard-parts-hydra.html
Details
Message ID
<87eenlxf0v.fsf@euandre.org>
In-Reply-To
<bab08df6-be8e-47bd-a254-7b86c41a9b47@www.fastmail.com> (view parent)
DKIM signature
pass
Download raw message
"Seph Gentle" <me@josephg.com> writes:

> This is worth watching if you haven’t seen it. Martin is at the forefront of the field - but this gives you a sense of where the research tech is at:
>
> https://martin.kleppmann.com/2020/07/06/crdt-hard-parts-hydra.html

Coincidentally, that's the  third item on my list 😬
Reply to thread Export thread (mbox)