~protesilaos/denote

6 2

Backlinks search has become slow recently

Details
Message ID
<87pma6t59i.fsf@kotlak.com>
DKIM signature
missing
Download raw message
Hi Prot,

I use "straight" package manager and track the latest "denote" from
"sr.ht". There are around 630 notes in my "denote-directory" and usually
the backlink search using "denote-links-find-backlinks" is blazing
fast. Backlinks show immediately. However, for the last few days I
noticed that it is being slow and taking around 2-3 seconds every time.
To debug the issue, I rolled back few commits and found that the
slowness is caused by these 2 commits:

https://git.sr.ht/~protesilaos/denote/commit/fa8c3e6600318abe86b4a6e18b4c95dc9dfba633
https://git.sr.ht/~protesilaos/denote/commit/9d048ba449f5cade28efb4a3e37d25bf506cc6bc


I have "xref-search-program" set to "ripgrep". I also verified that
directly running ripgrep on my "denote-directory" is fast (I also tried
the exacts commands run by denote/xref which I captured using "ps ax").

I will also try to go through the exact changes in the commits so that I
can be more helpful, however, I am just starting up with elisp so my
mileage may not be too much.

Thanks,


P.S. Sending first time to sr.ht mailing list, hopefully it shows up correctly.
Details
Message ID
<87ilfyynm1.fsf@protesilaos.com>
In-Reply-To
<87pma6t59i.fsf@kotlak.com> (view parent)
DKIM signature
missing
Download raw message
> From: utils+sr.ht@kotlak.com
> Date: Sun, 19 Feb 2023 13:33:55 +0530
>
> Hi Prot,

Hello there!

> I use "straight" package manager and track the latest "denote" from
> "sr.ht". There are around 630 notes in my "denote-directory" and usually
> the backlink search using "denote-links-find-backlinks" is blazing
> fast. Backlinks show immediately. However, for the last few days I
> noticed that it is being slow and taking around 2-3 seconds every time.
> To debug the issue, I rolled back few commits and found that the
> slowness is caused by these 2 commits:
>
> https://git.sr.ht/~protesilaos/denote/commit/fa8c3e6600318abe86b4a6e18b4c95dc9dfba633
> https://git.sr.ht/~protesilaos/denote/commit/9d048ba449f5cade28efb4a3e37d25bf506cc6bc

I expected this to happen.  It is a change I made in the hope of making
directory trees in silos behave as expected.  It still needs to be
optimised though.  I am interested in ideas on how best to approach this
issue.

> P.S. Sending first time to sr.ht mailing list, hopefully it shows up correctly.

Yes, it did.  Thank you!

All the best,
Prot

-- 
Protesilaos Stavrou
https://protesilaos.com
Details
Message ID
<87k00dc3tg.fsf@kotlak.com>
In-Reply-To
<87ilfyynm1.fsf@protesilaos.com> (view parent)
DKIM signature
missing
Download raw message
Protesilaos Stavrou <info@protesilaos.com> writes:

>> From: utils+sr.ht@kotlak.com
>> Date: Sun, 19 Feb 2023 13:33:55 +0530
>>
>> Hi Prot,
>
> Hello there!
>
>> I use "straight" package manager and track the latest "denote" from
>> "sr.ht". There are around 630 notes in my "denote-directory" and usually
>> the backlink search using "denote-links-find-backlinks" is blazing
>> fast. Backlinks show immediately. However, for the last few days I
>> noticed that it is being slow and taking around 2-3 seconds every time.
>> To debug the issue, I rolled back few commits and found that the
>> slowness is caused by these 2 commits:
>>
>> https://git.sr.ht/~protesilaos/denote/commit/fa8c3e6600318abe86b4a6e18b4c95dc9dfba633
>> https://git.sr.ht/~protesilaos/denote/commit/9d048ba449f5cade28efb4a3e37d25bf506cc6bc
>
> I expected this to happen.  It is a change I made in the hope of making
> directory trees in silos behave as expected.  It still needs to be
> optimised though.  I am interested in ideas on how best to approach this
> issue.
>

Ok, understood. May be we can have a "denote-silo-nesting-depth" value
of "0" to indicate we are only interested in the global
"denote-directory" and don't want to scan over multiple directories
trying to find a silo. What do you think?


>> P.S. Sending first time to sr.ht mailing list, hopefully it shows up correctly.
>
> Yes, it did.  Thank you!
>
> All the best,
> Prot

P.S. I think I sent the previous mail only to you and not to the
list. Re-sending it here.
Details
Message ID
<87o7ppoagz.fsf@protesilaos.com>
In-Reply-To
<87k00dc3tg.fsf@kotlak.com> (view parent)
DKIM signature
missing
Download raw message
> From: relict007 <utils+sr.ht@kotlak.com>
> Date: Sun, 19 Feb 2023 16:35:07 +0530

> [... 21 lines elided]

>> I expected this to happen.  It is a change I made in the hope of making
>> directory trees in silos behave as expected.  It still needs to be
>> optimised though.  I am interested in ideas on how best to approach this
>> issue.
>>
>
> Ok, understood. May be we can have a "denote-silo-nesting-depth" value
> of "0" to indicate we are only interested in the global
> "denote-directory" and don't want to scan over multiple directories
> trying to find a silo. What do you think?

That is an option, yes.  Another optimisation is for the
'denote--silo-p' which currently searches all 'directory-files'.  That
is wasteful.

At any rate, this is still disucssed on the GitHub mirror:
<https://github.com/protesilaos/denote/issues/129>.

There may be an easier way around this problem, in which case I will
remove the code I added and we willbe good to go.

-- 
Protesilaos Stavrou
https://protesilaos.com
Details
Message ID
<87bklpi80l.fsf@kotlak.com>
In-Reply-To
<87o7ppoagz.fsf@protesilaos.com> (view parent)
DKIM signature
missing
Download raw message
Protesilaos Stavrou <info@protesilaos.com> writes:

>> From: relict007 <utils+sr.ht@kotlak.com>
>> Date: Sun, 19 Feb 2023 16:35:07 +0530
>
>> [... 21 lines elided]
>
>>> I expected this to happen.  It is a change I made in the hope of making
>>> directory trees in silos behave as expected.  It still needs to be
>>> optimised though.  I am interested in ideas on how best to approach this
>>> issue.
>>>
>>
>> Ok, understood. May be we can have a "denote-silo-nesting-depth" value
>> of "0" to indicate we are only interested in the global
>> "denote-directory" and don't want to scan over multiple directories
>> trying to find a silo. What do you think?
>
> That is an option, yes.  Another optimisation is for the
> 'denote--silo-p' which currently searches all 'directory-files'.  That
> is wasteful.
>
> At any rate, this is still disucssed on the GitHub mirror:
> <https://github.com/protesilaos/denote/issues/129>.
>
> There may be an easier way around this problem, in which case I will
> remove the code I added and we willbe good to go.
Ohh ok. I will also check this out on Github.
Details
Message ID
<87zg98euu6.fsf@protesilaos.com>
In-Reply-To
<87bklpi80l.fsf@kotlak.com> (view parent)
DKIM signature
missing
Download raw message
> From: relict007 <utils+sr.ht@kotlak.com>
> Date: Mon, 20 Feb 2023 04:20:10 +0530

> [... 23 lines elided]

>> At any rate, this is still disucssed on the GitHub mirror:
>> <https://github.com/protesilaos/denote/issues/129>.
>>
>> There may be an easier way around this problem, in which case I will
>> remove the code I added and we willbe good to go.
> Ohh ok. I will also check this out on Github.

Notwithstanding the ongoing discussion, please try commit 3d2e1d5.  It
changes the following function to its current form:

    (defun denote--silo-p (path)
      "Return path to silo if PATH is a silo."
      (when-let (((and path (file-directory-p path)))
                 (dir-locals (dir-locals-find-file path)))
        (cond
         ((listp dir-locals)
          (car dir-locals))
         ((stringp dir-locals)
          dir-locals)
         (t nil))))

-- 
Protesilaos Stavrou
https://protesilaos.com
Details
Message ID
<87zg98c0kc.fsf@kotlak.com>
In-Reply-To
<87zg98euu6.fsf@protesilaos.com> (view parent)
DKIM signature
missing
Download raw message
Protesilaos Stavrou <info@protesilaos.com> writes:

>> From: relict007 <utils+sr.ht@kotlak.com>
>> Date: Mon, 20 Feb 2023 04:20:10 +0530
>
>> [... 23 lines elided]
>
>>> At any rate, this is still disucssed on the GitHub mirror:
>>> <https://github.com/protesilaos/denote/issues/129>.
>>>
>>> There may be an easier way around this problem, in which case I will
>>> remove the code I added and we willbe good to go.
>> Ohh ok. I will also check this out on Github.
>
> Notwithstanding the ongoing discussion, please try commit 3d2e1d5.  It
> changes the following function to its current form:
>
>     (defun denote--silo-p (path)
>       "Return path to silo if PATH is a silo."
>       (when-let (((and path (file-directory-p path)))
>                  (dir-locals (dir-locals-find-file path)))
>         (cond
>          ((listp dir-locals)
>           (car dir-locals))
>          ((stringp dir-locals)
>           dir-locals)
>          (t nil))))

I just sync'd the code and can confirm that finding backlinks
is fast again. Thanks Prot.
Reply to thread Export thread (mbox)