~rjarry/aerc-devel

4 3

[DRAFT PATCH aerc] aerc: add message index

Details
Message ID
<20220128115545.132129-1-vladimir@mgyar.me>
DKIM signature
pass
Download raw message
Patch: +205 -5
Added a new package `lib/msgindex` with leveldb implementation for
cacheing message headers by message UID and Message-ID.

Such a index would help in the future to implement threading in imap
(withouth IMAP THREAD support on the server) and in maildir. Also
fetching the headers in imap much faster.

IMAP and maildir workers open the index in the worker Configure message
and in the FetchMessageHeader they add the message to the index.

The index saves the headers by the key:

    headers_$(Message-ID) -> header data

and for finding the he headers by UID:

    uid_$(UID) -> $(Message-ID)

References: https://todo.sr.ht/~rjarry/aerc/2
Signed-off-by: Vladimír Magyar <vladimir@mgyar.me>
---
 go.mod                        |   1 +
 go.sum                        |  29 +++++++++
 lib/msgindex/index-leveldb.go | 117 ++++++++++++++++++++++++++++++++++
 lib/msgindex/msgindex.go      |  28 ++++++++
 worker/imap/fetch.go          |  21 ++++--
 worker/imap/worker.go         |   5 ++
 worker/maildir/worker.go      |   9 ++-
 7 files changed, 205 insertions(+), 5 deletions(-)
 create mode 100644 lib/msgindex/index-leveldb.go
 create mode 100644 lib/msgindex/msgindex.go

diff --git a/go.mod b/go.mod
index 3a18379..df2ed5c 100644
--- a/go.mod
+++ b/go.mod
@@ -31,6 +31,7 @@ require (
	github.com/pkg/errors v0.9.1
	github.com/riywo/loginshell v0.0.0-20200815045211-7d26008be1ab
	github.com/stretchr/testify v1.4.0
	github.com/syndtr/goleveldb v1.0.1-0.20210819022825-2ae1ddf74ef7
	github.com/zenhack/go.notmuch v0.0.0-20211022191430-4d57e8ad2a8b
	golang.org/x/crypto v0.0.0-20211215153901-e495a2d5b3d3 // indirect
	golang.org/x/net v0.0.0-20211029224645-99673261e6eb // indirect
diff --git a/go.sum b/go.sum
index 00e4036..6b8abb7 100644
--- a/go.sum
+++ b/go.sum
@@ -82,6 +82,8 @@ github.com/envoyproxy/go-control-plane v0.9.0/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymF
github.com/envoyproxy/go-control-plane v0.9.1-0.20191026205805-5f8ba28d4473/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4=
github.com/envoyproxy/go-control-plane v0.9.4/go.mod h1:6rpuAdCZL397s3pYoYcLgu1mIlRU8Am5FuJP05cCM98=
github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7+kN2VEUnK/pcBlmesArF7c=
github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo=
github.com/fsnotify/fsnotify v1.4.9/go.mod h1:znqG4EE+3YCdAaPaxE2ZRY/06pZUdp0tY4IgpuI1SZQ=
github.com/fsnotify/fsnotify v1.5.1 h1:mZcQUHVQUQWoPXXtuf9yuEXKudkV2sx1E06UadKWpgI=
github.com/fsnotify/fsnotify v1.5.1/go.mod h1:T3375wBYaZdLLcVNkcVbzGHY7f1l/uK5T5Ai1i3InKU=
github.com/gdamore/encoding v1.0.0 h1:+7OoQ1Bc6eTm5niUzBa0Ctsh6JbMW6Ra+YNuAtDBdko=
@@ -120,6 +122,8 @@ github.com/golang/protobuf v1.4.2/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw
github.com/golang/protobuf v1.5.0/go.mod h1:FsONVRAS9T7sI+LIUmWTfcYkHO4aIWwzhcaSAoJOfIk=
github.com/golang/protobuf v1.5.2 h1:ROPKBNFfQgOUMifHyP+KYbvpjbdoFNs+aK7DXlji0Tw=
github.com/golang/protobuf v1.5.2/go.mod h1:XVQd3VNwM+JqD3oG2Ue2ip4fOMUkwXdXDdiuN0vRsmY=
github.com/golang/snappy v0.0.4 h1:yAGX7huGHXlcLOEtBnF4w7FQwA26wojNCwOYAEhLjQM=
github.com/golang/snappy v0.0.4/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q=
github.com/google/btree v0.0.0-20180813153112-4030bb1f1f0c/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ=
github.com/google/btree v1.0.0/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ=
github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M=
@@ -147,6 +151,7 @@ github.com/googleapis/gax-go/v2 v2.0.4/go.mod h1:0Wqv26UfaUD9n4G6kQubkQ+KchISgw+
github.com/googleapis/gax-go/v2 v2.0.5/go.mod h1:DWXyrwAJ9X0FpwwEdw+IPEYBICEFu5mhpdKc/us6bOk=
github.com/hashicorp/golang-lru v0.5.0/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8=
github.com/hashicorp/golang-lru v0.5.1/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8=
github.com/hpcloud/tail v1.0.0/go.mod h1:ab1qPbhIpdTxEkNHXyeSf5vhxWSCs/tWer42PpOxQnU=
github.com/ianlancetaylor/demangle v0.0.0-20181102032728-5e5cf60278f6/go.mod h1:aSSvb/t6k1mPoxDqO4vJh6VOCGPwU4O0C2/Eqndh1Sc=
github.com/imdario/mergo v0.3.12 h1:b6R2BslTbIEToALKP7LxUvijTsNI9TAe80pLWN2g/HU=
github.com/imdario/mergo v0.3.12/go.mod h1:jmQim1M+e3UYxmgPu/WyfjB3N3VflVyUjjjwH0dnCYA=
@@ -176,6 +181,15 @@ github.com/miolini/datacounter v1.0.2 h1:mGTL0vqEAtH7mwNJS1JIpd6jwTAP6cBQQ2P8apa
github.com/miolini/datacounter v1.0.2/go.mod h1:C45dc2hBumHjDpEU64IqPwR6TDyPVpzOqqRTN7zmBUA=
github.com/mitchellh/go-homedir v1.1.0 h1:lukF9ziXFxDFPkA1vsr5zpc1XuPDn/wFntq5mG+4E0Y=
github.com/mitchellh/go-homedir v1.1.0/go.mod h1:SfyaCUpYCn1Vlf4IUYiD9fPX4A5wJrkLzIz1N1q0pr0=
github.com/nxadm/tail v1.4.4 h1:DQuhQpB1tVlglWS2hLQ5OV6B5r8aGxSrPc5Qo6uTN78=
github.com/nxadm/tail v1.4.4/go.mod h1:kenIhsEOeOJmVchQTgglprH7qJGnHDVpk1VPCcaMI8A=
github.com/onsi/ginkgo v1.6.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
github.com/onsi/ginkgo v1.12.1/go.mod h1:zj2OWP4+oCPe1qIXoGWkgMRwljMUYCdkwsT2108oapk=
github.com/onsi/ginkgo v1.14.0 h1:2mOpI4JVVPBN+WQRa0WKH2eXR+Ey+uK4n7Zj0aYpIQA=
github.com/onsi/ginkgo v1.14.0/go.mod h1:iSB4RoI2tjJc9BBv4NKIKWKya62Rps+oPG/Lv9klQyY=
github.com/onsi/gomega v1.7.1/go.mod h1:XdKZgCCFLUoM/7CFJVPcG8C1xQ1AJ0vpAezJrB7JYyY=
github.com/onsi/gomega v1.10.1 h1:o0+MgICZLuZ7xjH7Vx6zS/zcu93/BEp1VwkIW1mEXCE=
github.com/onsi/gomega v1.10.1/go.mod h1:iN09h71vgCQne3DLsj+A5owkum+a2tYe+TOCB1ybHNo=
github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
@@ -191,6 +205,8 @@ github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
github.com/stretchr/testify v1.4.0 h1:2E4SXV/wtOkTonXsotYi4li6zVWxYlZuYNCXe9XRJyk=
github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4=
github.com/syndtr/goleveldb v1.0.1-0.20210819022825-2ae1ddf74ef7 h1:epCh84lMvA70Z7CTTCmYQn2CKbY8j86K7/FAIr141uY=
github.com/syndtr/goleveldb v1.0.1-0.20210819022825-2ae1ddf74ef7/go.mod h1:q4W45IWZaF22tdD+VEXcAWRA037jwmWEB5VWYORlTpc=
github.com/yuin/goldmark v1.1.25/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
github.com/yuin/goldmark v1.1.32/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
@@ -232,6 +248,7 @@ golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20190108225652-1e06a53dbb7e/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20190213061140-3a22650c66bd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
@@ -252,9 +269,11 @@ golang.org/x/net v0.0.0-20200324143707-d3edc9973b7e/go.mod h1:qpuaurCH72eLCgpAm/
golang.org/x/net v0.0.0-20200501053045-e0ff5e5a1de5/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A=
golang.org/x/net v0.0.0-20200506145744-7e3656a0809f/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A=
golang.org/x/net v0.0.0-20200513185701-a91f0712d120/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A=
golang.org/x/net v0.0.0-20200520004742-59133d7f0dd7/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A=
golang.org/x/net v0.0.0-20200520182314-0ba52f642ac2/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A=
golang.org/x/net v0.0.0-20200625001655-4c5254603344/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
golang.org/x/net v0.0.0-20200707034311-ab3426394381/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
golang.org/x/net v0.0.0-20200813134508-3edf25e44fcc/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
golang.org/x/net v0.0.0-20200822124328-c89045814202/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
golang.org/x/net v0.0.0-20211029224645-99673261e6eb h1:pirldcYWx7rx7kE5r+9WsOXPXK0+WH5+uZ7uPmJ44uM=
golang.org/x/net v0.0.0-20211029224645-99673261e6eb/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
@@ -274,6 +293,7 @@ golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJ
golang.org/x/sync v0.0.0-20200317015054-43a5402ce75a/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20200625203802-6e8e738ad208/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190312061237-fead79001313/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
@@ -282,7 +302,10 @@ golang.org/x/sys v0.0.0-20190507160741-ecd444e8653b/go.mod h1:h1NjWce9XRLGQEsW7w
golang.org/x/sys v0.0.0-20190606165138-5da285871e9c/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190624142023-c5567b49c5d0/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190726091711-fc99dfbffb4e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190904154756-749cb33beabd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20191001151750-bb3f8db39f24/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20191005200804-aed5e4c7ecf9/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20191120155948-bd437916bb0e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20191204072324-ce4227a45e2e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20191228213918-04cbcbbfeed8/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200113162924-86b910548bc1/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
@@ -296,8 +319,10 @@ golang.org/x/sys v0.0.0-20200331124033-c3d80250170d/go.mod h1:h1NjWce9XRLGQEsW7w
golang.org/x/sys v0.0.0-20200501052902-10377860bb8e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200511232937-7e40ca221e25/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200515095857-1151b9dac4a9/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200519105757-fe76b779f299/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200523222454-059865788121/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200803210538-64077c9b5642/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200814200057-3d37ad5750ed/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210423082822-04245dca01da/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
@@ -447,7 +472,11 @@ gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8
gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127 h1:qIbj1fsPNlZgppZ+VLlY7N33q108Sa+fhmuc+sWQYwY=
gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/errgo.v2 v2.1.0/go.mod h1:hNsd1EY+bozCKY1Ytp96fpM3vjJbqLJn88ws8XvfDNI=
gopkg.in/fsnotify.v1 v1.4.7/go.mod h1:Tz8NjZHkW78fSQdbUxIjBTcgA1z1m8ZHf0WmKUhAMys=
gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7 h1:uRGJdciOHaEIrze2W8Q3AKkepLTh2hOroT7a+7czfdQ=
gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7/go.mod h1:dt/ZhP58zS4L8KSrWDmTeBkI65Dw0HsyUHuEVlX15mw=
gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.2.4/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.3.0 h1:clyUAQHOM3G0M3f5vQj7LuJrETvjVot3Z5el9nffUtU=
gopkg.in/yaml.v2 v2.3.0/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
diff --git a/lib/msgindex/index-leveldb.go b/lib/msgindex/index-leveldb.go
new file mode 100644
index 0000000..f99eeb7
--- /dev/null
+++ b/lib/msgindex/index-leveldb.go
@@ -0,0 +1,117 @@
package msgindex

import (
	"bufio"
	"bytes"
	"fmt"
	"os"
	"path/filepath"

	"github.com/emersion/go-message/textproto"
	"github.com/syndtr/goleveldb/leveldb"
)

func NewIndexLevelDB(accountName string) (Interface, error) {
	home, err := os.UserHomeDir()
	if err != nil {
		return nil, fmt.Errorf("could not resolve home directory: %v", err)
	}
	path := filepath.Join(home, ".cache", "aerc", accountName)
	db, err := leveldb.OpenFile(path, nil)
	if err != nil {
		return nil, fmt.Errorf("opening msg index: %w", err)
	}
	return &indexLevelDB{db: db}, nil
}

type indexLevelDB struct {
	db *leveldb.DB
}

//adds two pairs into db
//uid_$uid -> $msgID
//headers_$msgID -> header data
func (index *indexLevelDB) Add(uid uint32, msgID string, header textproto.Header) error {
	k := uidKey(uid)
	if has, err := index.db.Has(k, nil); err != nil {
		return err
	} else if has {
		return nil
	}
	var headerData bytes.Buffer
	if err := textproto.WriteHeader(&headerData, header); err != nil {
		return fmt.Errorf("msg index add - reading header data: %w", err)
	}
	batch := &leveldb.Batch{}
	batch.Put(k, []byte(msgID))
	batch.Put(msgIDKey(msgID), headerData.Bytes())
	if err := index.db.Write(batch, nil); err != nil {
		return fmt.Errorf("msg index add - writing: %w", err)
	}
	return nil
}

func (index *indexLevelDB) GetByUID(uid uint32) (*textproto.Header, error) {
	k := uidKey(uid)
	msgID, err := index.db.Get(k, nil)
	if err != nil {
		if err == leveldb.ErrNotFound {
			return nil, ErrNotFound
		}
		return nil, fmt.Errorf("msg index get by uid: %w", err)
	}
	return index.GetByMessageID(string(msgID))
}

func (index *indexLevelDB) GetByMessageID(msgID string) (*textproto.Header, error) {
	headerData, err := index.db.Get(msgIDKey(msgID), nil)
	if err != nil {
		if err == leveldb.ErrNotFound {
			return nil, ErrNotFound
		}
		return nil, fmt.Errorf("msg index get by msgID: %w", err)
	}
	header, err := textproto.ReadHeader(bufio.NewReader(bytes.NewBuffer(headerData)))
	if err != nil {
		return nil, fmt.Errorf("msg index parsing header: %w", err)
	}
	return &header, nil
}

func (index *indexLevelDB) Delete(uid uint32) error {
	k := uidKey(uid)
	msgID, err := index.db.Get(k, nil)
	if err != nil {
		if err == leveldb.ErrNotFound {
			return ErrNotFound
		}
		return fmt.Errorf("msg index delete: %w", err)
	}
	batch := &leveldb.Batch{}
	batch.Delete(k)
	batch.Delete(msgIDKey(string(msgID)))
	if err := index.db.Write(batch, nil); err != nil {
		return fmt.Errorf("msg index delete - writing: %w", err)
	}
	return nil
}

func (index *indexLevelDB) Close() error {
	return index.db.Close()
}

func uidKey(uid uint32) []byte {
	return append(
		[]byte("uid_"),
		[]byte{
			byte(uid),
			byte(uid >> 8),
			byte(uid >> 16),
			byte(uid >> 24),
		}...,
	)
}

func msgIDKey(msgID string) []byte {
	return append([]byte("headers_"), []byte(msgID)...)
}
diff --git a/lib/msgindex/msgindex.go b/lib/msgindex/msgindex.go
new file mode 100644
index 0000000..ec19910
--- /dev/null
+++ b/lib/msgindex/msgindex.go
@@ -0,0 +1,28 @@
package msgindex

import (
	"errors"

	"github.com/emersion/go-message/textproto"
)

var ErrNotFound error = errors.New("msg not found")

type Interface interface {
	//adds new message headers to the index
	//if the uid exits, don't rewrite
	Add(uid uint32, msgID string, header textproto.Header) error

	//gets message headers by uid
	GetByUID(uid uint32) (*textproto.Header, error)

	//gets message headers by Message-ID
	GetByMessageID(msgID string) (*textproto.Header, error)

	//deletes message headers from the index
	//deletes also the MessageID
	Delete(uid uint32) error

	//closes the index
	Close() error
}
diff --git a/worker/imap/fetch.go b/worker/imap/fetch.go
index c63ee42..e31f2f1 100644
--- a/worker/imap/fetch.go
+++ b/worker/imap/fetch.go
@@ -10,6 +10,7 @@ import (
	"github.com/emersion/go-message/mail"
	"github.com/emersion/go-message/textproto"

	"git.sr.ht/~rjarry/aerc/lib/msgindex"
	"git.sr.ht/~rjarry/aerc/models"
	"git.sr.ht/~rjarry/aerc/worker/types"
)
@@ -35,10 +36,22 @@ func (imapw *IMAPWorker) handleFetchMessageHeaders(
	}
	imapw.handleFetchMessages(msg, msg.Uids, items,
		func(_msg *imap.Message) error {
			reader := _msg.GetBody(section)
			textprotoHeader, err := textproto.ReadHeader(bufio.NewReader(reader))
			if err != nil {
				return fmt.Errorf("could not read header: %v", err)
			var textprotoHeader textproto.Header
			if h, err := imapw.index.GetByMessageID(_msg.Envelope.MessageId); err == nil {
				textprotoHeader = *h
			} else {
				if err != msgindex.ErrNotFound {
					return fmt.Errorf("could not read msg index: %w", err)
				} else {
					reader := _msg.GetBody(section)
					textprotoHeader, err = textproto.ReadHeader(bufio.NewReader(reader))
					if err != nil {
						return fmt.Errorf("could not read header: %v", err)
					}
					if err := imapw.index.Add(_msg.Uid, _msg.Envelope.MessageId, textprotoHeader); err != nil {
						return fmt.Errorf("could not add msg to index: %w", err)
					}
				}
			}
			header := &mail.Header{message.Header{textprotoHeader}}
			imapw.worker.PostMessage(&types.MessageInfo{
diff --git a/worker/imap/worker.go b/worker/imap/worker.go
index ba53df2..d30b269 100644
--- a/worker/imap/worker.go
+++ b/worker/imap/worker.go
@@ -15,6 +15,7 @@ import (
	"golang.org/x/oauth2"

	"git.sr.ht/~rjarry/aerc/lib"
	"git.sr.ht/~rjarry/aerc/lib/msgindex"
	"git.sr.ht/~rjarry/aerc/models"
	"git.sr.ht/~rjarry/aerc/worker/handlers"
	"git.sr.ht/~rjarry/aerc/worker/types"
@@ -58,6 +59,7 @@ type IMAPWorker struct {
	seqMap        []uint32
	done          chan struct{}
	autoReconnect bool
	index         msgindex.Interface
}

func NewIMAPWorker(worker *types.Worker) (types.Backend, error) {
@@ -169,6 +171,8 @@ func (w *IMAPWorker) handleMessage(msg types.WorkerMessage) error {
				w.config.keepalive_interval = int(val.Seconds())
			}
		}
		//open msg index
		w.index, reterr = msgindex.NewIndexLevelDB(msg.Config.Name)
	case *types.Connect:
		if w.client != nil && w.client.State() == imap.SelectedState {
			reterr = fmt.Errorf("Already connected")
@@ -221,6 +225,7 @@ func (w *IMAPWorker) handleMessage(msg types.WorkerMessage) error {
			break
		}
		w.autoReconnect = false
		w.index.Close()
		w.worker.PostMessage(&types.Done{types.RespondTo(msg)}, nil)
	case *types.ListDirectories:
		w.handleListDirectories(msg)
diff --git a/worker/maildir/worker.go b/worker/maildir/worker.go
index 2f75f12..821e65f 100644
--- a/worker/maildir/worker.go
+++ b/worker/maildir/worker.go
@@ -13,6 +13,7 @@ import (
	"github.com/emersion/go-maildir"
	"github.com/fsnotify/fsnotify"

	"git.sr.ht/~rjarry/aerc/lib/msgindex"
	"git.sr.ht/~rjarry/aerc/models"
	"git.sr.ht/~rjarry/aerc/worker/handlers"
	"git.sr.ht/~rjarry/aerc/worker/lib"
@@ -33,6 +34,7 @@ type Worker struct {
	worker              *types.Worker
	watcher             *fsnotify.Watcher
	currentSortCriteria []*types.SortCriterion
	index               msgindex.Interface
}

// NewWorker creates a new maildir worker with the provided worker.
@@ -236,7 +238,8 @@ func (w *Worker) handleConfigure(msg *types.Configure) error {
	}
	w.c = c
	w.worker.Logger.Printf("configured base maildir: %s", dir)
	return nil
	w.index, err = msgindex.NewIndexLevelDB(msg.Config.Name)
	return err
}

func (w *Worker) handleConnect(msg *types.Connect) error {
@@ -386,6 +389,10 @@ func (w *Worker) handleFetchMessageHeaders(
			w.err(msg, err)
			continue
		}
		if err := w.index.Add(uid, info.Envelope.MessageId, info.RFC822Headers.Header.Header); err != nil {
			w.worker.Logger.Printf("could not add message to index: %v", err)
			w.err(msg, err)
		}
		w.worker.PostMessage(&types.MessageInfo{
			Message: types.RespondTo(msg),
			Info:    info,
-- 
2.35.0
Details
Message ID
<CHHLDMWR1DNG.9R911AD7J0BV@diabtop>
In-Reply-To
<20220128115545.132129-1-vladimir@mgyar.me> (view parent)
DKIM signature
missing
Download raw message
Vladimír Magyar, Jan 28, 2022 at 12:55:
> Added a new package `lib/msgindex` with leveldb implementation for
> cacheing message headers by message UID and Message-ID.

s/cacheing/caching/

> Such a index would help in the future to implement threading in imap
> (withouth IMAP THREAD support on the server) and in maildir. Also
> fetching the headers in imap much faster.

s/withouth/without/

s/in imap much faster/in imap is much faster/

> IMAP and maildir workers open the index in the worker Configure message
> and in the FetchMessageHeader they add the message to the index.
>
> The index saves the headers by the key:
>
>     headers_$(Message-ID) -> header data
>
> and for finding the he headers by UID:
>
>     uid_$(UID) -> $(Message-ID)

For now, I don't think we need this for maildir (not until we think
about threading support).

Also, could you please split the implementation of the index/cache and
its use for imap in two separate patches?

> diff --git a/lib/msgindex/index-leveldb.go b/lib/msgindex/index-leveldb.go
> new file mode 100644
> index 0000000..f99eeb7
> --- /dev/null
> +++ b/lib/msgindex/index-leveldb.go
> @@ -0,0 +1,117 @@
> +package msgindex

Maybe "headersindex" is more suitable? That would leave the possibility
to cache message bodies as well for imap later on.

> +func NewIndexLevelDB(accountName string) (Interface, error) {
> +	home, err := os.UserHomeDir()
> +	if err != nil {
> +		return nil, fmt.Errorf("could not resolve home directory: %v", err)
> +	}
> +	path := filepath.Join(home, ".cache", "aerc", accountName)

To leave some room for the body cache, could you please store the
headers index into "~/.cache/aerc/$account/headers/"?

> +	db, err := leveldb.OpenFile(path, nil)

This seems to create missing directories automatically which is nice.

> +	if err != nil {
> +		return nil, fmt.Errorf("opening msg index: %w", err)
> +	}
> +	return &indexLevelDB{db: db}, nil
> +}
> +
> +type indexLevelDB struct {
> +	db *leveldb.DB
> +}

> diff --git a/worker/imap/fetch.go b/worker/imap/fetch.go
> index c63ee42..e31f2f1 100644
> --- a/worker/imap/fetch.go
> +++ b/worker/imap/fetch.go
> @@ -10,6 +10,7 @@ import (
>  	"github.com/emersion/go-message/mail"
>  	"github.com/emersion/go-message/textproto"
>  
> +	"git.sr.ht/~rjarry/aerc/lib/msgindex"
>  	"git.sr.ht/~rjarry/aerc/models"
>  	"git.sr.ht/~rjarry/aerc/worker/types"
>  )
> @@ -35,10 +36,22 @@ func (imapw *IMAPWorker) handleFetchMessageHeaders(
>  	}
>  	imapw.handleFetchMessages(msg, msg.Uids, items,
>  		func(_msg *imap.Message) error {
> -			reader := _msg.GetBody(section)
> -			textprotoHeader, err := textproto.ReadHeader(bufio.NewReader(reader))
> -			if err != nil {
> -				return fmt.Errorf("could not read header: %v", err)
> +			var textprotoHeader textproto.Header
> +			if h, err := imapw.index.GetByMessageID(_msg.Envelope.MessageId); err == nil {

I wonder how _msg.Envelope.MessageId is available here. Shouldn't you
use index.GetByUID() instead?

I'm not sure the index is used here. The problem is that
imapw.handleFetchMessages already calls:

	if err := imapw.client.UidFetch(set, items, messages); err != nil {

With items containing:

		imap.FetchBodyStructure,
		imap.FetchEnvelope,
		imap.FetchInternalDate,
		imap.FetchFlags,
		imap.FetchUid,
		section.FetchItem(),

This inline callback function is invoked for each response from the imap
server. We would save tons of time if all these fields were already
stored in the index (under _msg.Uid) and less message envelopes/flags
would need to be fetched.

> +				textprotoHeader = *h
> +			} else {
> +				if err != msgindex.ErrNotFound {
> +					return fmt.Errorf("could not read msg index: %w", err)
> +				} else {
> +					reader := _msg.GetBody(section)
> +					textprotoHeader, err = textproto.ReadHeader(bufio.NewReader(reader))
> +					if err != nil {
> +						return fmt.Errorf("could not read header: %v", err)
> +					}
> +					if err := imapw.index.Add(_msg.Uid, _msg.Envelope.MessageId, textprotoHeader); err != nil {
> +						return fmt.Errorf("could not add msg to index: %w", err)
> +					}
> +				}
>  			}
>  			header := &mail.Header{message.Header{textprotoHeader}}
>  			imapw.worker.PostMessage(&types.MessageInfo{
> diff --git a/worker/imap/worker.go b/worker/imap/worker.go
> index ba53df2..d30b269 100644
> --- a/worker/imap/worker.go
> +++ b/worker/imap/worker.go
> @@ -15,6 +15,7 @@ import (
>  	"golang.org/x/oauth2"
>  
>  	"git.sr.ht/~rjarry/aerc/lib"
> +	"git.sr.ht/~rjarry/aerc/lib/msgindex"
>  	"git.sr.ht/~rjarry/aerc/models"
>  	"git.sr.ht/~rjarry/aerc/worker/handlers"
>  	"git.sr.ht/~rjarry/aerc/worker/types"
> @@ -58,6 +59,7 @@ type IMAPWorker struct {
>  	seqMap        []uint32
>  	done          chan struct{}
>  	autoReconnect bool
> +	index         msgindex.Interface
>  }
>  
>  func NewIMAPWorker(worker *types.Worker) (types.Backend, error) {
> @@ -169,6 +171,8 @@ func (w *IMAPWorker) handleMessage(msg types.WorkerMessage) error {
>  				w.config.keepalive_interval = int(val.Seconds())
>  			}
>  		}
> +		//open msg index
> +		w.index, reterr = msgindex.NewIndexLevelDB(msg.Config.Name)
>  	case *types.Connect:
>  		if w.client != nil && w.client.State() == imap.SelectedState {
>  			reterr = fmt.Errorf("Already connected")
> @@ -221,6 +225,7 @@ func (w *IMAPWorker) handleMessage(msg types.WorkerMessage) error {
>  			break
>  		}
>  		w.autoReconnect = false
> +		w.index.Close()

index.Close() should not be done in disconnect but when shutting now
the imap worker. Otherwise you will run into issues when reconnecting.

>  		w.worker.PostMessage(&types.Done{types.RespondTo(msg)}, nil)
>  	case *types.ListDirectories:
>  		w.handleListDirectories(msg)
> diff --git a/worker/maildir/worker.go b/worker/maildir/worker.go
> index 2f75f12..821e65f 100644
> --- a/worker/maildir/worker.go
> +++ b/worker/maildir/worker.go
> @@ -13,6 +13,7 @@ import (
>  	"github.com/emersion/go-maildir"
>  	"github.com/fsnotify/fsnotify"
>  
> +	"git.sr.ht/~rjarry/aerc/lib/msgindex"
>  	"git.sr.ht/~rjarry/aerc/models"
>  	"git.sr.ht/~rjarry/aerc/worker/handlers"
>  	"git.sr.ht/~rjarry/aerc/worker/lib"
> @@ -33,6 +34,7 @@ type Worker struct {
>  	worker              *types.Worker
>  	watcher             *fsnotify.Watcher
>  	currentSortCriteria []*types.SortCriterion
> +	index               msgindex.Interface
>  }
>  
>  // NewWorker creates a new maildir worker with the provided worker.
> @@ -236,7 +238,8 @@ func (w *Worker) handleConfigure(msg *types.Configure) error {
>  	}
>  	w.c = c
>  	w.worker.Logger.Printf("configured base maildir: %s", dir)
> -	return nil
> +	w.index, err = msgindex.NewIndexLevelDB(msg.Config.Name)
> +	return err
>  }
>  
>  func (w *Worker) handleConnect(msg *types.Connect) error {
> @@ -386,6 +389,10 @@ func (w *Worker) handleFetchMessageHeaders(
>  			w.err(msg, err)
>  			continue
>  		}
> +		if err := w.index.Add(uid, info.Envelope.MessageId, info.RFC822Headers.Header.Header); err != nil {
> +			w.worker.Logger.Printf("could not add message to index: %v", err)
> +			w.err(msg, err)
> +		}

This index is not used yet in maildir. No need to spend CPU time
building it. Please remove for now.

I believe this would warrant something in the docs to explain what is
cached and where. Probably in aerc-imap(5).

Thanks.
Details
Message ID
<CHHZ1OETCC85.3BIN2ZL7ZF5FE@micro-pc>
In-Reply-To
<CHHLDMWR1DNG.9R911AD7J0BV@diabtop> (view parent)
DKIM signature
fail
Download raw message
DKIM signature: fail
> For now, I don't think we need this for maildir (not until we think
> about threading support).

Making It for maildir (and then using it for threading) was my entire
motivation to do this path. So I would like to leave it there, or just
disable it, when threading for maildir is false. That would need to
change where the `ThreadingEnabled` flag is. We must move it from
`UIConfig` to `AccountConfig`. But I'll leave it to another patch.

> Also, could you please split the implementation of the index/cache and
> its use for imap in two separate patches?

Yes I'll do it. Can I send it right away?

> > diff --git a/lib/msgindex/index-leveldb.go b/lib/msgindex/index-leveldb.go
> > new file mode 100644
> > index 0000000..f99eeb7
> > --- /dev/null
> > +++ b/lib/msgindex/index-leveldb.go
> > @@ -0,0 +1,117 @@
> > +package msgindex
>
> Maybe "headersindex" is more suitable? That would leave the possibility
> to cache message bodies as well for imap later on.

I don't think this is a good idea. Then, when you want to cache the
bodies, there will be another implementation with name "bodyindex"?
Thought that making the `MsgIndex` as an Interface and the
`indexLevelDB` as it's implementation. So that when a better kv store
could be used, it can be easily rewritten.

> > +func NewIndexLevelDB(accountName string) (Interface, error) {
> > +	home, err := os.UserHomeDir()
> > +	if err != nil {
> > +		return nil, fmt.Errorf("could not resolve home directory: %v", err)
> > +	}
> > +	path := filepath.Join(home, ".cache", "aerc", accountName)
>
> To leave some room for the body cache, could you please store the
> headers index into "~/.cache/aerc/$account/headers/"?

I don't know if it has any benefits to split the database so much, but
can't find anything that proves it.

> > +			if h, err := imapw.index.GetByMessageID(_msg.Envelope.MessageId); err == nil {
>
> I wonder how _msg.Envelope.MessageId is available here. Shouldn't you
> use index.GetByUID() instead?

> I'm not sure the index is used here. The problem is that
> imapw.handleFetchMessages already calls:
>
> 	if err := imapw.client.UidFetch(set, items, messages); err != nil {

You're right, if we want to use the cache, we eider store the entire
MesasgeInfo (worker/imap/fetch.go#L59) or figure out what items of these
are not needed:

> 		imap.FetchBodyStructure,
> 		imap.FetchEnvelope,
> 		imap.FetchInternalDate,
> 		imap.FetchFlags,
> 		imap.FetchUid,
> 		section.FetchItem(),

I would go for storing `models.MessageInfo`.

> This inline callback function is invoked for each response from the imap
> server. We would save tons of time if all these fields were already
> stored in the index (under _msg.Uid) and less message envelopes/flags
> would need to be fetched.

Yes we need to add `index.Add` to the callback and check the index
before wee call `imapw.handleFetchMessages`

> index.Close() should not be done in disconnect but when shutting now
> the imap worker. Otherwise you will run into issues when reconnecting.

Where is the shutting down part of the worker?

> I believe this would warrant something in the docs to explain what is
> cached and where. Probably in aerc-imap(5).

When I know if we will store MessageInfo or just the headers I'll do it
then.
Details
Message ID
<CHI1CNI93XE0.O3YP4J06QGYS@MacBook-Air.local>
In-Reply-To
<20220128115545.132129-1-vladimir@mgyar.me> (view parent)
DKIM signature
pass
Download raw message
On Fri Jan 28, 2022 at 7:55 PM +08, Vladimír Magyar wrote:
> +	home, err := os.UserHomeDir()
> +	if err != nil {
> +		return nil, fmt.Errorf("could not resolve home directory: %v", err)
> +	}
> +	path := filepath.Join(home, ".cache", "aerc", accountName)

Maybe use xdg.CacheHome() instead of ~/.cache? You can get that by
importing "github.com/kyoh86/xdg" (it's already used in other parts of
aerc). This would be ~/Library/Caches on Mac, and ~/.cache on other OSs.

For example

+path := filepath.Join(xdg.ConfigHome(), "aerc", accountName)

instead of the 5 lines I quoted.
Details
Message ID
<CHI22MSZKTY1.ZF3SWV2GPHW9@micro-pc>
In-Reply-To
<CHI1CNI93XE0.O3YP4J06QGYS@MacBook-Air.local> (view parent)
DKIM signature
pass
Download raw message
That's great! I'll use that, thanks.
Reply to thread Export thread (mbox)