I use aerc with maildir to keep local copies on my machine. I download
the mail via IMAP using mbsync[0] (isync) program. I use Fastmail as
my email provider. I have been using this setup succesfully for more
than a year.
I decided to reorganize some of my emails today. I created nested folders
via my provider's web app and synced via mbsync immediately.
I started moving the messages into different folders by dozen. I think
eventually there were few hundred emails moved this way.
When I was finished I decided to sync, the sync was succesful, I could
see messages in my other folders on my iOS Mail app.
But I noticed something unexpected immediately. The messages were not
sorted properly in iOS Mail app. All those moved messages had timestamp
from today, actually it looked like the timestamp of a time when
I moved the message between folders (!). It doesn't look right and it
makes looking up messages in my folders unintuitive.
I noticed that this problem is not visible in aerc. The messages in
the message list show the correct timestamp, when I open a message
the "date" field shows the correct timestamp as well.
I did raise my problem in mbsync mailing list[1]. I wasn't completely
sure whether this problem happened because of the sync.
Later I went to IRC channel for aerc. One of the channel users
managed to recreate the same problem, the user also used Fastmail:
> 2022-08-15 17:43:57 rockorager comatory: I can recreate that issue (also on fastmail)> 2022-08-15 17:48:48 rockorager Fastmail isn't changing any headerrs> 2022-08-15 17:49:31 rockorager Strange...> 2022-08-15 17:50:39 rockorager I'll check out the Jmap data soon
I disconnected few minutes after that so not sure if the user managed
to dig into it deeper but hopefully he'll notice this mail and perhaps
provide more information.
So I'm unsure where the problem could be originating from, it seems
like it's connected to Fastmail service.
This is what my configuration for aerc looks like (I removed my
domain and username, substituted):
```
source = maildir://~/mail/account.com/
outgoing = smtp+plain://user%40account.com@smtp.fastmail.com
outgoing-cred-cmd = gpg --decrypt --no-tty ~/mail/account.com.gpg
default = INBOX
smtp-starttls = yes
from = User Name <user@account.com>
copy-to = Sent
```
If it's possible I would like other email clients to start showing
the correct dates, most of the time I'm using aerc so at least I have
that but I'm often dealing with my mail in other clients as well.
I'd be grateful for any help. I can help with debugging if I'm given
some instructions on how to do that. I do have go toolchain installed
as I'm building aerc from source but I'm not a go programmer.
Cheers,
Ondrej
---
[0]: https://sourceforge.net/projects/isync/
[1]: https://sourceforge.net/p/isync/mailman/isync-devel/thread/CM6OO6TIN3QW.J6FUZV4Q48G6%40Ondrej-Synacek-YYG7RKYP6F.local/#msg37693793
On Mon Aug 15, 2022 at 1:20 PM CDT, Ondřej Synáček wrote:
> I did raise my problem in mbsync mailing list[1]. I wasn't completely> sure whether this problem happened because of the sync.> Later I went to IRC channel for aerc. One of the channel users> managed to recreate the same problem, the user also used Fastmail:>> > 2022-08-15 17:43:57 rockorager comatory: I can recreate that issue (also on fastmail)> > 2022-08-15 17:48:48 rockorager Fastmail isn't changing any headerrs> > 2022-08-15 17:49:31 rockorager Strange...> > 2022-08-15 17:50:39 rockorager I'll check out the Jmap data soon
Hi - FYI I am rockorager on IRC.
> I'd be grateful for any help. I can help with debugging if I'm given> some instructions on how to do that. I do have go toolchain installed> as I'm building aerc from source but I'm not a go programmer.
I looked into the data structures and here is what's happening:
1. mbsync gives the imap server the time of file creation for the server
to store as INTERNAL DATE. The server *does not need* to honor this, but
fastmail does.
2. Apple Mail and the fastmail interface use the INTERNAL DATE value for
displaying and sorting mail. You can see this in aerc by sorting by
either `sort -r arrival` (uses INTERNAL DATE) and `sort -r date` (uses
DATE from the envelope). You will need a patch applied if you are
running off master to restore sorting functionality - I just sent it a
little bit ago.
3. aerc uses the Date from the envelope to display the timestamp.
4. aerc doesn't "move" messages, it copies and deletes. I was actually
working on a patch to change this for imap yesterday, and ran into a ton
of difficulties implementing this in maildir because of the file
watcher. The maildir move command implements an os level file rename,
which retains the creation date - this would solve this in aerc-land.
Still, fastmail doesn't *have* to honor the creation date method. But,
they do based on my limited testing just now.
Could you start a ticket referencing this issue?
Tim
On Tue Aug 16, 2022 at 6:50 PM CEST, Tim Culverhouse wrote:
> 1. mbsync gives the imap server the time of file creation for the server> to store as INTERNAL DATE. The server *does not need* to honor this, but> fastmail does.
Yes it does seem to be the case. I just opened ticket in Fastmail and
got in touch with the tech support. It seems like they confirm this (quotes):
> Basically each message has two dates/time associated with it:>> 1. The sent date/time, which is what's in the Date: header> 2. The received date/time, which is stored internally in the server, and set> when the message is delivered to the mailbox (internaldate)> For email uploaded from an external source via IMAP, the uploading program is> able to set the "internaldate" when it uploads, so in this case, it appears the> wrong date was set when it uploaded the message.
On Tue Aug 16, 2022 at 6:50 PM CEST, Tim Culverhouse wrote:
> 2. Apple Mail and the fastmail interface use the INTERNAL DATE value for> displaying and sorting mail. You can see this in aerc by sorting by> either `sort -r arrival` (uses INTERNAL DATE) and `sort -r date` (uses> DATE from the envelope).
Yep that seems to be the case.
On Tue Aug 16, 2022 at 6:50 PM CEST, Tim Culverhouse wrote:
> 4. [...] I was actually working on a patch to change this for imap yesterday,> and ran into a ton > of difficulties implementing [...]> Could you start a ticket referencing this issue?
So just to check this, this seems to be aerc-related or does it also
involve mbsync? Or is it both?
Sure I'll open a ticket. I guess this is the URL to do it?
https://todo.sr.ht/~rjarry/aerc
Thanks a bunch!
Hey I wanted to thank ~rockorager for the patch[1].
I tried moving messages between the folders now and I'm running into another
problem - it might probably be better to create another ticket for this.
Or maybe you can explain to me whether this behaviour is somehow intentional.
I moved some messages from my inbox to folder (and also one of them into
a subfolder). Whenever I sync with mbsync now I get error message for
each of these messages (I assume)
```
C: 0/1 B: 17/30 F: +0/0 *1/1 #0/0 N: +0/0 *0/0 #0/0
Maildir error: UID 376 is beyond highest assigned UID 227.
C: 0/1 B: 21/30 F: +0/0 *1/1 #0/0 N: +0/0 *0/0 #0/0
Maildir error: UID 374 is beyond highest assigned UID 101.
C: 1/1 B: 30/30 F: +0/0 *1/1 #0/0 N: +0/0 *0/0 #0/0
```
I can still see the messages I moved in their respective folders but when
opening the mail in other clients (webmail, iOS Mail app), the message
now disappeared.
It seems like whatever message I move out of inbox now disappears in other
clients :(
[1] https://git.sr.ht/~rjarry/aerc/commit/c98f70487417825ca592c928cbf0144ba88eef7e
On Tue Aug 23, 2022 at 2:15 AM EDT, Ondřej Synáček wrote:
> I moved some messages from my inbox to folder (and also one of them into> a subfolder). Whenever I sync with mbsync now I get error message for> each of these messages (I assume)>> ```> C: 0/1 B: 17/30 F: +0/0 *1/1 #0/0 N: +0/0 *0/0 #0/0> Maildir error: UID 376 is beyond highest assigned UID 227.> C: 0/1 B: 21/30 F: +0/0 *1/1 #0/0 N: +0/0 *0/0 #0/0> Maildir error: UID 374 is beyond highest assigned UID 101.> C: 1/1 B: 30/30 F: +0/0 *1/1 #0/0 N: +0/0 *0/0 #0/0> ```
I also noticed similar errors with my mbsync setup today, e.g.
> Maildir error: duplicate UID 2415.
and
> Maildir error: UID 6060 is beyond highest assigned UID 2417.
That commit (c98f7048) does seem to be the cause -- I removed it and
recompiled, and now moving between folders works as usual. As for
getting my maildir back in order to prevent the mbsync errors, manually
renaming the mail files as roughly described here[0] did the trick. In
short, remove the ",U=<number>" from the filename of one of the two
files with the duplicate UID (while leaving intact trailing flags such
as ":2,S").
[0]: http://tiborsimko.org/mbsync-duplicate-uid.html
On Tue Aug 23, 2022 at 9:03 AM CEST, Thomas Faughnan wrote:
> That commit (c98f7048) does seem to be the cause -- I removed it and> recompiled, and now moving between folders works as usual.
Thanks Thomas for letting me know, at least it confirms that the
specific patch causes it.
I'm not sure what to do at the moment, my timestamps are getting messed
up but that's still probably better than not being able to see
the messages in other clients at all.
I will not revert to previous build of aerc yet just in case
maintainers need more detailed information that I could provide with
my setup. Let's see this will depend on response in this thread.
Sorry for the bug!
It looks like it should be an easy fix...instead of using the builtin
maildir move, we can manually create the new path+filename and - if it
has a trailing ,U=\d* - remove that portion of the key.
I'll try to have something out today for that.
Tim
On Tue Aug 23, 2022 at 2:51 PM CEST, Tim Culverhouse wrote:
> Sorry for the bug!
Hey no worries, as long as it doesn't delete the messages I think
that's the price to pay for living with upstream build.
Appreciate the work on it!
> It looks like it should be an easy fix...instead of using the builtin> maildir move, we can manually create the new path+filename and - if it> has a trailing ,U=\d* - remove that portion of the key.
Won't that cause the problem with changed timestamps again? Because I
believe the problem was the move wasn't actually move but copy.
I don't know, I don't have enough context for the code, just wanted to
point it out.
Either way, I already created a ticket: https://todo.sr.ht/~rjarry/aerc/75
Thanks a lot!
On Tue Aug 23, 2022 at 7:57 AM CDT, Ondřej Synáček wrote:
> Won't that cause the problem with changed timestamps again? Because I> believe the problem was the move wasn't actually move but copy.> I don't know, I don't have enough context for the code, just wanted to> point it out.
Nope, shouldn't be an issue! The built-in maildir move basically does:
mv /path/to/email,U=123:S /newpath/to/email,U=123:S
I can implement separately in aerc to do:
mv /path/to/email,U=123:S /newpath/to/email:S
It's the same operation, just a different destination for the move.
Tim