dispatch.sr.ht overhaul proposal

Details
Message ID
<BUTMP0M9HDGV.16DDFWUTXGJTE@homura>
Sender timestamp
1560532759
DKIM signature
pass
Download raw message
Hiya! I've been thinking about dispatch.sr.ht lately. Now that most of
sr.ht's APIs are fleshed out, it's time to start thinking about how to
get them talking to each other more. The goal for some time now has been
to use dispatch.sr.ht, but I'm not satisfied with the current design.

So, here's a different design. Opening the floor to discussion. Let me
start by enumerating some use-cases:

  *.sr.ht
    N git repos
    N trackers
    1 mailing list (sr.ht-dev)
  aerc
    1 git repo, 1 mailing list, 1 tracker
  scdoc
    1 git repo, shares mailing list with others, 1 tracker
  git-send-email.io
    1 git repo, shares mailing list with others
  sway
    1 git repo on GitHub, uses builds.sr.ht for builds

With these, I want to be able to, for example:

- Send patches sent to a mailing list to CI, ideally associating them
  with a specific git repo in the case of shared lists
- Ditto, but for mailing lists on external services
- Link ticket mentions in emails, commits, etc, to git repos
- Sync tickets between todo.sr.ht and GitHub/GitLab/etc
- Sync GitLab MRs/GitHub PRs with mailing list threads and vice versa
- Import git repos, tickets, etc from other services

The current approach of implementing each specific use-case in its own
job, one at a time, is not scalable to handling all of these use-cases
and all of the options I want for customization. To address this, I
would like to move dispatch.sr.ht to a YAML-based configuration file.

Consider the following primitives:

- Sources (e.g. a GitHub repo, lists.sr.ht mailing list, etc)
- Events (e.g. a git commit, an email to the list)
- Sinks (e.g. builds.sr.ht/submit)

Sources emit events, which you can route to a sink.

Your dispatch graph can be configured with a YAML file which lists
routes:

  - name: forward changelog
    source: git.sr.ht/~sircmpwn/*.sr.ht/push/tag
    sink: lists.sr.ht/~sircmpwn/sr.ht-announce

  - name: push to build
    source: git.sr.ht/~sircmpwn/*.sr.ht/push/*
    sink: builds.sr.ht/submit
    secrets: true

  - name: patch to build
    source: lists.sr.ht/~sircmpwn/sr.ht-dev/email/patch
    sink: builds.sr.ht/submit
    secrets: false
    email-results: true

  - name: link commits to tickets
    source: git.sr.ht/~sircmpwn/*.sr.ht/push/*
    sink: todo.sr.ht/~sircmpwn/$repo_name

  - name: link tickets to commits
    source: todo.sr.ht/~sircmpwn/*.sr.ht
    sink: git.sr.ht/~sircmpwn/$tracker_name

You can store this file in a git repo anywhere and point dispatch.sr.ht
to it. It could be a branch of some project, or a standalone repo
defining a graph for a number of projects, or a .dispatch.yml file
sitting on your master branch. I would also consider letting you upload
YAML files to dispatch.sr.ht on the fly (which will also be useful for
migrating existing GitHub rules painlessly).

In terms of implementation, sources would implement a number of
interfaces. Things like:

- Commit
- Ticket
- Email
- Patch

Sources emit events which implement one or more of these interfaces, and
sinks have a list of the kinds of events they understand. For example,
the builds.sr.ht sink might understand patches and commits.  Patches
could come from lists.sr.ht, commits from git.sr.ht or GitHub, etc. It
could also support more specific events, e.g. "GitHubPullRequest",
explicitly.

Some other considerations:

- Eventually the top-level project hub will be configuring these for you
  automatically when you create a new project. So instead of (1) create
  repo, (2) create tracker, (3) create list, (4) rig up builds on
  dispatch, (5) rig up ticket references, (etc), it'll be: (1) fill out
  project details and accept the sane defaults, then change them later
  if you need to. Therefore, these files need to be machine editable.
- This format would also be fairly easy to update on the web, to avoid
  confronting users with a scary YAML file and a bunch of stuff to learn
  upfront. We can both render the graph as HTML and edit it with some
  simple forms.

Whenever you try to update the graph, we'll do some quick validation.
If, for example, we need you to authorize us to access your GitHub
account, we'll push back with an error and a URL to correct it.

Feedback welcome.