Weekly status update (2020-19)

Message ID
DKIM signature
Download raw message
Hey there,

I've wanted to get into the habit of writing down things I've done week
be week, to give people an insight into what is being done, how the
project is progressing, and also to ask general questions that I don't
know the answer to, but that someone else might be very experienced in.

If you have questions about any of the points raised in the update, feel
free to just reply to the list, or join our weekly voice call (Monday
16:00 UTC, check the list for the link!)

Tcp overlay network

The first thing I worked on was the tcp overlay for ratman, called
"netmod-tcp".  The way that ratman (which I've now dubbed to mean
Routing And Transmission MANager) is implemented means that it can't
have knowledge about device specific endpoints (drivers).  It can route
messages across a network, but doesn't know (or care) if the underlying
medium is Tcp, an in-memory channel, some phone bluetooth stuff, or a
wet shoelace.

Therefore a lot of the logic needs to sit in the actualy endpoint
drivers.  We now have three working endpoints: mem, udp, and tcp.  The
way the latter works is by taking a list of peers, that the driver will
try to connect to.  Once "peered", it can then route packages to nodes
behind that address.

The way this works is that ratman gets information about what interface
and "target id" a user announcement came from, and can then direct the
drivers to target specific targets.  This way we can implement
one-to-many interfaces.

The current netmod-tcp implementation is pretty simple: it will retry to
peer if a peer is offline, but won't re-connect if the connection is
lost.  This is something that should be improved on.  And generally it's
internal design isn't as elegant as it could be.  Ideally we also want
the tcp driver to do "dynamic peering", meaning that based on some
criteria (which are yet to be bikeshedded), a peer should be able to
introduce other peers to each other to increase the density of

libqaul and virtual network testing

The next thing I tackled was testing!  We have a lot of code in our
repository, and a lot of it was untested.  It was kinda awkward to
create tests too, because just bootstrapping a virtual network would
take maybe 50 lines of code.  Doing this in every test is annoying,
replicating the same test harness between different crates almost more

So I wrote a small crate: ratman-harness.  Currently it only does one
thing: create a three-node network with the topology A - B - C, while
giving direct access to A and C via getters, that can then be used to
send messages around.  The middle node is purely there to re-transmit
packages and doesn't need to run an application stack.

In the future I think it would be cool to have ratman-harness be able to
generate larger networks, while keeping the same API of exposing only 2
(or a few more discrete) nodes via functions that can be used to
actually simulate user behaviour.  The rest of the network could then
represent different topologies to test how the services handle excessive
delay, loops, etc.

Ideally this is something that could be done with quickcheck, but I
haven't spent too much time thinking about that.

Configuring ratman: ratman-configure

The next problem I tackled was configuring the router!  While ratman
itself is pretty simple to configure, it needs endpoints to hook into to
make it useful (otherwise it's basically just an async glorified
/dev/null).  A problem also arose with many clients having to do this
setup again and again.  How could we make it simpler to use ratman, in
third party code, but also our own applications?

The idea is a configuration crate which exposes a json schema that can
be used to create a ratman instance with pre-configured endpoints.
Currently this supports the in-memory network (why I'm not sure but it
was simple enough to make work), udp and tcp endpoints.  In the future
we'll add wifi-direct for android, maybe bluetooth, and whatever else
people want to see.

It would also be nice to have a macro DLS for rust because constructing
recursive structs is annoying, and json in Rust even more so.  But this
is a good start for now.  Again, if this seems like something you'd have
fun writing some code for, please get in touch!

Persistent routing daemon: qauld

Another idea we had a few weeks ago was having a qaul daemon that could
be run on a system to handle routing and application state, so that
potential UIs didn't have to.  It also meant that a UI could crash and
the router could keep running.  Potentially this would allow qauld to
act as a general userspace VPN endpoint.

I took some of the code that Nora had written a while ago and adapted
it.  It's now using ratman-configure, but it's not exposing any rpc
interfaces yet, so it's not yet usable.  Hopefully I'll get it to work
properly in the next few days or so.

qaul.net chat service

Last but not least, the chat service!  It sits on top of libqaul,
handles text messages between multiple people, in DMs or chat-rooms.  It
was mostly implementede already (from ~1 week ago), but it had some
pretty bad bugs.  Some of them were fixed by Mathias, and some I
literally just got working.  It'll need a lot more testing, and
unfortunately there's some load-bearing sleep statements in the tests
that can't be avoided right now.

(The problem is that events need to have some time to propagate through
the network before we can act on the information that we shouldn't have,
because it technically hasn't arrived yet.  Ideally we would have some
sort of gate-system here, but again, not something I've spent too much
time thinking about yet)

And that's it!  As I said, I'll try to write these every sunday, just
before our weekly voice call.  Feel free to ask questions if you have
them, or if you want to help out with something, that's much appreciated
to.  Cheers!