I have encountered some issue sending a patch series to lists.sr.ht

It first happened when 2 days ago a patch series was incomplete[0] and
the last part didn't show up (my attempt to just forward the patch from
my mailbox certainly didn't help).

That time the issue was "fixed" by just sending it again, but that's not
really a solution. Since it happened again today[1] I was curious if
other people have experienced something similar. I tried sending the
same message again (I wrote down the headers this time) and it was just
dropped without any response again.

My mailserver logs don't show any error[2].

[0]: https://lists.sr.ht/~rjarry/aerc-devel/%3C20220618101917.399233-1-moritz%40poldrack.dev%3E
[1]: https://lists.sr.ht/~rjarry/aerc-devel/%3C20220620082222.196328-1-moritz%40poldrack.dev%3E
[2]: https://paste.sr.ht/~poldi1405/ceabb2fc2e86379fbebf1011961a78fe7152a4ab
--
Moritz Poldrack
https://moritz.sh

I just received word from one of the mailinglist subscribers that they
did receive 4/4. Even less of an explanation as to why it doesn't show
on the webUI

Another funny detail: It's included in the mbox export:
https://lists.sr.ht/~rjarry/aerc-devel/%3C20220620082222.196328-1-moritz%40poldrack.dev%3E/mbox
--
Moritz Poldrack
https://moritz.sh

On Mon Jun 20, 2022 at 9:47 PM +0200, Moritz Poldrack wrote:
> Another funny detail: It's included in the mbox export:
> https://lists.sr.ht/~rjarry/aerc-devel/%3C20220620082222.196328-1-moritz%40poldrack.dev%3E/mbox

FWIW it's not limited to patches, I posted this in sr.ht-dev
two months ago <CCJJWBWPYJRHH.P27EGATOLHJR@nix> and no one
seemed to know why either.

On Tue Apr 26, 2022 at 1:45 PM KST, Nguyễn Gia Phong wrote:
> Case in point: [0] vs [1].  This message is also available
> in the archive at [2].  I wonder if this has been reported before
> and if I should file a ticket for it.
>
> [0] https://lists.sr.ht/~cnx/blackshades/<e4f806dc-96b6-403d-92c4-bf534834e3a2@icculus.org>/raw
> [1] https://lists.sr.ht/~cnx/blackshades/<e4f806dc-96b6-403d-92c4-bf534834e3a2@icculus.org>
> [2] https://lists.sr.ht/~cnx/blackshades

On Tue Jun 21, 2022 at 6:02 AM CEST, Nguyễn Gia Phong wrote:
> FWIW it's not limited to patches, I posted this in sr.ht-dev
> two months ago <CCJJWBWPYJRHH.P27EGATOLHJR@nix> and no one
> seemed to know why either.

Glad to know I'm not the only one, but not great that we don't know the
cause…

I am trying to reproduce this locally by importing the mbox files, but 
so far I cannot, everything looks just fine.

Moritz, can you confirm though that this actually looks allright? I 
couldn't see anything missing there:

https://lists.sr.ht/~rjarry/aerc-devel/%3C20220620082222.196328-1-moritz%40poldrack.dev%3E

As for 
https://lists.sr.ht/~cnx/blackshades/<e4f806dc-96b6-403d-92c4-bf534834e3a2@icculus.org>, 
I have no clue so far. Renders fine in my local instance. I'll keep 
poking it...

Cheers,
Conrad

On Wed Jun 22, 2022 at 1:29 PM CEST, Conrad Hoffmann wrote:
> I am trying to reproduce this locally by importing the mbox files, but
> so far I cannot, everything looks just fine.
>
> Moritz, can you confirm though that this actually looks allright? I
> couldn't see anything missing there:
>
> https://lists.sr.ht/~rjarry/aerc-devel/%3C20220620082222.196328-1-moritz%40poldrack.dev%3E

I looks like this now has all mails listed which it originally didn't.
Might be due to Tim's reply (just a guess though). One telling indicator
that something's off is that it's not recognised as a patch though and
that the CI pipeline was not triggert, which it usually is.

On Wed Jun 22, 2022 at 6:48 AM CDT, Moritz Poldrack wrote:
> I looks like this now has all mails listed which it originally didn't.
> Might be due to Tim's reply (just a guess though). One telling indicator
> that something's off is that it's not recognised as a patch though and
> that the CI pipeline was not triggert, which it usually is.

This doesn't show v4 4/4 for me. The messages for me show as:

[PATCH aerc v4 0/4]
[PATCH aerc v4 3/4]
[PATCH aerc v4 1/4]
[PATCH aerc v4 2/4]
Re: [PATCH aerc v4 4/4]

On 6/22/22 14:21, Tim Culverhouse wrote:
> On Wed Jun 22, 2022 at 6:48 AM CDT, Moritz Poldrack wrote:
>> One telling indicator that something's off is that it's not
>> recognised as a patch

Indeed.

> This doesn't show v4 4/4 for me.

Ack, I over-read the "Re:" of the 4/4 there.

Thanks, it seems there is more at play than just the specific emails. 
Both issues do not appear on my local import of the mbox file.

I'll dig further and will let you know if I find s/t

Quick update on this: we've identified one problem with thread 
assignment where mails arrive out of order. There is code to handle 
this, but it had some issues. A patch is in review. Note, however, that 
the patch only prevents this from happening in the future. We'll also 
need to write a script to fix the already broken threads (WIP).

This is definitely what happened with the Blackshades thread. Patches 
have a slightly different handling, I still need to check whether that 
was caused by the same issue or a different one.

Thanks for your patience, y'all. I'll keep you posted.

Cheers,
Conrad

On 6/24/22 19:11, Conrad Hoffmann wrote:
> This is definitely what happened with the Blackshades thread. Patches 
> have a slightly different handling, I still need to check whether that 
> was caused by the same issue or a different one.

Well, turns out the reordering issue also fixes the patch detection. 
Even though I find that a bit curious, I'll just accept it and instead 
focus on writing something that fixes the already broken threads in the 
database.

Cheers,
Conrad

On Fri Jun 24, 2022 at 7:11 PM +0200, Conrad Hoffmann wrote:
> Quick update on this: we've identified one problem with thread 
> assignment where mails arrive out of order. [...] This is definitely
> what happened with the Blackshades thread.

May I ask which order it is?  I want to resort the mbox
and test around a little.

> I'll keep you posted.

Thanks!

On 6/29/22 06:22, Nguyễn Gia Phong wrote:
> May I ask which order it is?  I want to resort the mbox
> and test around a little.

For sure. The message IDs, in the order that they were inserted into the 
database:

9D6CFBFE-7191-48D0-9D50-9B6486D44929@zacharyjackslater.com
CANkt0ZcKHrn-H073TQqObkK6AbEnZ-Dc4eYXiqUq+7mB0D4qPA@mail.gmail.com
e4f806dc-96b6-403d-92c4-bf534834e3a2@icculus.org
CBZTB9S4OY9Y.IUCWJ8TO80NJ@nix
CC5KX31G7Y9G.FUEV6FB9UG99@nix

If you're curious, here is the patch that I submitted: 
https://lists.sr.ht/~sircmpwn/sr.ht-dev/patches/33251

Happy to report that a fix has been rolled out and we ran a query to fix 
the database. Both reported threads now look as expected.

Note that we did _not_ re-run the patch detection, that would be a bit 
more involved. But the fix put in place will at least make the patch 
detection work for future patches that arrive out of order.

Happy hacking,
Conrad

On 6/29/22 06:22, Nguyễn Gia Phong wrote:
> On Fri Jun 24, 2022 at 7:11 PM +0200, Conrad Hoffmann wrote:
>> Quick update on this: we've identified one problem with thread
>> assignment where mails arrive out of order. [...] This is definitely
>> what happened with the Blackshades thread.
> 
> May I ask which order it is?  I want to resort the mbox
> and test around a little.
> 
>> I'll keep you posted.
> 
> Thanks!

On Fri Jul 1, 2022 at 11:10 AM +0200, Conrad Hoffmann wrote:
> Happy to report that a fix has been rolled out and we ran a query
> to fix the database. Both reported threads now look as expected.

Thanks!  BTW the count over https://lists.sr.ht/~cnx/blackshades
still does not reflect the number if displayed threads.
Is it just caching or another issue to be ironed out?