Hi folks,

I know that many of us use scripts like msmtpq to queue messages that
fail to send and automatically retry sending them later. I personally
find most of these scripts lacking in terms of integration with the mail
client and have my own custom script that improves the experience a bit
for me by indexing the send queue in notmuch so that I can at least view
it in aerc. I'm contemplating building some send queue functionality
directly into aerc, as described below, and welcome any feedback you
have.

To begin, here are my goals for the send queue's behavior:

- When a message fails to send, queue it to be sent later, regardless of
  the outgoing configuration.
- Automatically attempt to send queued messages in the background while
  aerc is running. Ideally this could also be done without aerc open
  with some sort of background script/command.
- Allow viewing and deleting messages in the queue from within aerc.
- Don't copy a message to sent until sending succeeds.
- (maybe) Allow queueing messages that were composed outside of aerc,
  e.g. by git send-email.
- (maybe) Allow converting queued messages to drafts so that they can be
  edited before sending.
- (maybe) Allow scheduling message send, e.g. after 30 seconds (to allow
  "undoing" send), or at 8am tomorrow, etc.

The send queue would of course be optional and configurable.

I have a rough idea of how this functionality might be implemented as
well. The send queue is stored as a maildir on disk, one per account,
regardless of the account backend type. If sending a message fails, it
is placed in the queue maildir. An external program can also queue a
message to be sent simply by placing it in this maildir.

A background goroutine is used to periodically attempt to re-send any
messages in the queue maildir. They are sent using the outgoing
configuration for their account. Ideally, a command line flag allows
running only this queue flushing code so that external scripts can flush
the queue on demand even if aerc is not open; I'm not sure how hard to
implement this flag would be, though.

For the base functionality, no extra information needs to be stored with
the queued messages. However, to schedule messages to be sent later,
aerc adds a special header (e.g. X-Aerc-Send-After) indicating the
earliest time at which the message can be sent, and then the header
value is read when re-sending queued messages to ensure a message isn't
sent too soon. This header is removed when actually sending the message.

To allow viewing and deleting queued messages, add a special send queue
folder in the folder list. This folder shows all messages from the queue
maildir and allows for a subset of the usual message list commands, plus
maybe a few extras like converting a message to a draft.

I think the trickiest part of the implementation would be the use of a
maildir with non-maildir backends. Obviously aerc already has the
ability to work with maildirs, but only for maildir accounts. I'm not
sure yet what would be the cleanest way to use a maildir from
non-maildir accounts.

I'm not sure when I'll actually getting around to coding all this up --
it may be a few months -- but I'd like to get some feedback in the
meantime. Is this functionality desirable? Do you have ideas to improve
my proposed implementation? Am I missing anything?

Best,
Jason

Hey Jason,

On Sat Aug 24, 2024 at 03:40, Jason Cox <me@jasoncarloscox.com> wrote:
> Hi folks,
>
> I know that many of us use scripts like msmtpq to queue messages that
> fail to send and automatically retry sending them later. I personally
> find most of these scripts lacking in terms of integration with the mail
> client and have my own custom script that improves the experience a bit
> for me by indexing the send queue in notmuch so that I can at least view
> it in aerc. I'm contemplating building some send queue functionality
> directly into aerc, as described below, and welcome any feedback you
> have.

I think this is a great idea. If aerc could be functionally a complete
replacement for msmptq that would be great.

>
> To begin, here are my goals for the send queue's behavior:
>
> - When a message fails to send, queue it to be sent later, regardless of
>   the outgoing configuration.
> - Automatically attempt to send queued messages in the background while
>   aerc is running. Ideally this could also be done without aerc open
>   with some sort of background script/command.

I think this would be very important, especially if you want the scheduling you
mention later, so people can plug this into a systemd timer without needing to
have aerc open.

> - Allow viewing and deleting messages in the queue from within aerc.
> - Don't copy a message to sent until sending succeeds.
> - (maybe) Allow queueing messages that were composed outside of aerc,
>   e.g. by git send-email.
> - (maybe) Allow converting queued messages to drafts so that they can be
>   edited before sending.
> - (maybe) Allow scheduling message send, e.g. after 30 seconds (to allow
>   "undoing" send), or at 8am tomorrow, etc.

Undo send actually has a ticket somewhere and scheduling has been asked about
a couple of times. I've contemplated modifying msmtpq to do this, so having it
in aerc would be superior. 

>
> The send queue would of course be optional and configurable.
>
> I have a rough idea of how this functionality might be implemented as
> well. The send queue is stored as a maildir on disk, one per account,
> regardless of the account backend type. If sending a message fails, it
> is placed in the queue maildir. An external program can also queue a
> message to be sent simply by placing it in this maildir.

I _really_ like the maildir idea, as it opens up additional scripting
possibilities.

>
> A background goroutine is used to periodically attempt to re-send any
> messages in the queue maildir. They are sent using the outgoing
> configuration for their account. Ideally, a command line flag allows
> running only this queue flushing code so that external scripts can flush
> the queue on demand even if aerc is not open; I'm not sure how hard to
> implement this flag would be, though.

As noted, I think this would be a pretty important capability.

>
> For the base functionality, no extra information needs to be stored with
> the queued messages. However, to schedule messages to be sent later,
> aerc adds a special header (e.g. X-Aerc-Send-After) indicating the
> earliest time at which the message can be sent, and then the header
> value is read when re-sending queued messages to ensure a message isn't
> sent too soon. This header is removed when actually sending the message.
>
> To allow viewing and deleting queued messages, add a special send queue
> folder in the folder list. This folder shows all messages from the queue
> maildir and allows for a subset of the usual message list commands, plus
> maybe a few extras like converting a message to a draft.
>
> I think the trickiest part of the implementation would be the use of a
> maildir with non-maildir backends. Obviously aerc already has the
> ability to work with maildirs, but only for maildir accounts. I'm not
> sure yet what would be the cleanest way to use a maildir from
> non-maildir accounts.
>
> I'm not sure when I'll actually getting around to coding all this up --
> it may be a few months -- but I'd like to get some feedback in the
> meantime. Is this functionality desirable? Do you have ideas to improve
> my proposed implementation? Am I missing anything?

I think in general replacing msmtpq with an aerc --flush-queue running on
a small local maildir folder is a great idea!

Best,
Bence

+1 to the proposal, it'd be a nice addition.

On 24/08/2024, 03:40, Jason Cox wrote:
> - (maybe) Allow converting queued messages to drafts so that they can be
>   edited before sending.

This is precisely what you get with :recall -f, am I right?

Regards,