Hi Prot,
I have written a small package to cache Denote files' metadata in memory
to help me with my specific use-cases.
https://git.sr.ht/~relict007/denote-cache
This package scans through all Denote notes and files and makes an
in-memory cache of their metadata including links for easy retrieval and
sorting etc. For this, I am using standard Emacs and Denote functions
that you have mentioned in "For developers or advanced users" section in
Denote manual. However, I could not find an easy way to retrieve forward
and back links using just the functions that you have mentioned
there. For now I have just copied the code from Denote source but that
means using internal functions (specially in getting forward links)
which are subject to change.
I wanted to know what your thought are on adding those public functions
to Denote which will help us avoid using anything internal. Or if you
think we should use xref/regular-expressions directly to fetch the
links.
This is my present code (mostly copied from Denote):
(defun denote-cache--retrieve-backlinks (file)
"Retrieve backlinks using denote native apis. No cache"
(setq identifier (denote-retrieve-filename-identifier file))
(delete file (sort
(cl-remove-duplicates
(mapcar #'xref-location-group
(mapcar #'xref-match-item-location
(xref-matches-in-files identifier (denote-directory-text-only-files)))) :test 'equal)
#'string-lessp)))
(defun denote-cache--retrieve-forwardlinks (file)
"Retrieve forward links, no cache"
(setq file-type (denote-filetype-heuristics file))
(if (eq file-type 'org)
(let* (
(regexp (denote--link-in-context-regexp file-type))
(files (denote-link--expand-identifiers regexp)))
(delete file files)
)
'()
)
)
thanks
Peter Prevos <peter@prevos.net> writes:
> Hi relict007,
Hi Peter,
>> Thanks for writing these caching functions, could be useful in my> efforts to visualise Denote networks.
You are welcome. Glad to be of help.
>> Regards>> P:)
> From: relict007 <utils+sr.ht@kotlak.com>> Date: Sun, 7 May 2023 11:20:53 +0530>> Hi Prot,
Hello there!
> I have written a small package to cache Denote files' metadata in memory> to help me with my specific use-cases.>> https://git.sr.ht/~relict007/denote-cache
Congratulations! What is your plan with this? Maybe add it to some
package archive? We can include it in the Denote manual as well.
> This package scans through all Denote notes and files and makes an> in-memory cache of their metadata including links for easy retrieval and> sorting etc. For this, I am using standard Emacs and Denote functions> that you have mentioned in "For developers or advanced users" section in> Denote manual. However, I could not find an easy way to retrieve forward> and back links using just the functions that you have mentioned> there. For now I have just copied the code from Denote source but that> means using internal functions (specially in getting forward links)> which are subject to change.> [... 34 lines elided]
I will check the details tomorrow. This is just an early comment. The
idea is to provide public functions. That way we cut down on
duplication and potential conflicts.
More to follow.
Prot
--
Protesilaos Stavrou
https://protesilaos.com
Protesilaos Stavrou <info@protesilaos.com> writes:
>> From: relict007 <utils+sr.ht@kotlak.com>>> Date: Sun, 7 May 2023 11:20:53 +0530>>>> Hi Prot,>> Hello there!
Hi Prot,
>>> I have written a small package to cache Denote files' metadata in memory>> to help me with my specific use-cases.>>>> https://git.sr.ht/~relict007/denote-cache>> Congratulations! What is your plan with this? Maybe add it to some> package archive? We can include it in the Denote manual as well.
Yes, I would like to add this to "gnu elpa" or otherwise to melpa. As
this is my first Emacs package, I am not familiar with the listing
process or their rules. I need to do some reading on that.
It is fairly usable and I have been using it for some time now, however,
before others start using it I would like to document it in detail
including its configuration options, public functions, limitations,
known issues, intended use cases etc. Also, as it doesn't provide any
user facing functionality in itself, documentation needs few ready to
use examples of its usage in various scenarios. Once it is at a
satisfactory level, I will send a separate "new package announcement"
kind of mail to Denote mailing list. And I will be very happy if its get
mentioned in the Denote manual.
>>> This package scans through all Denote notes and files and makes an>> in-memory cache of their metadata including links for easy retrieval and>> sorting etc. For this, I am using standard Emacs and Denote functions>> that you have mentioned in "For developers or advanced users" section in>> Denote manual. However, I could not find an easy way to retrieve forward>> and back links using just the functions that you have mentioned>> there. For now I have just copied the code from Denote source but that>> means using internal functions (specially in getting forward links)>> which are subject to change.>>> [... 34 lines elided]>> I will check the details tomorrow. This is just an early comment. The> idea is to provide public functions. That way we cut down on> duplication and potential conflicts.>> More to follow.
Sounds good.
>> Prot
thanks
> From: relict007 <utils+sr.ht@kotlak.com>> Date: Sun, 7 May 2023 20:14:59 +0530> [... 14 lines elided]>> I will check the details tomorrow. This is just an early comment. The>> idea is to provide public functions. That way we cut down on>> duplication and potential conflicts.>>>> More to follow.>> Sounds good.
As promised, I made the requisite changes and pushed them to 'main'
branch.
commit 694c1517be73949edbc3993c105c764da8e2571f
Author: Protesilaos Stavrou <info@protesilaos.com>
Date: Mon May 8 18:12:55 2023 +0300
Expose public functions for returning links or backlinks
We are doing the work in standalone functions that can be called from
Lisp, instead of limiting it to interactive functions. The code is
more composable as a result.
This is in response to the mailing list thread started by relict007:
<https://lists.sr.ht/~protesilaos/denote/%3C87a5ygk6yi.fsf@kotlak.com%3E>
relict007 is the developer of the 'denote-cache' package (in
progress): <https://git.sr.ht/~relict007/denote-cache>.
README.org | 10 ++++++++++
denote.el | 46 +++++++++++++++++++++++++++++++---------------
2 files changed, 41 insertions(+), 15 deletions(-)
Please test the functions 'denote-link-return-links', 'denote-link-return-backlinks'.
> [... 12 lines elided]>>> I have written a small package to cache Denote files' metadata in memory>>> to help me with my specific use-cases.>>>>>> https://git.sr.ht/~relict007/denote-cache>>>> Congratulations! What is your plan with this? Maybe add it to some>> package archive? We can include it in the Denote manual as well.>> Yes, I would like to add this to "gnu elpa" or otherwise to melpa. As> this is my first Emacs package, I am not familiar with the listing> process or their rules. I need to do some reading on that.>> It is fairly usable and I have been using it for some time now, however,> before others start using it I would like to document it in detail> including its configuration options, public functions, limitations,> known issues, intended use cases etc. Also, as it doesn't provide any> user facing functionality in itself, documentation needs few ready to> use examples of its usage in various scenarios. Once it is at a> satisfactory level, I will send a separate "new package announcement"> kind of mail to Denote mailing list. And I will be very happy if its get> mentioned in the Denote manual.
Okay, I can help you with the packaging. To put it on GNU ELPA you need
to assign copyright to the Free Software Foundation. This means that
you have to share your identity with them (though not with me or this
mailing list). MELPA has no such requirement.
I attach a few patches. They are not exhaustive, but should give you an
idea of how to tweak the code to meet the expected style.
There are three cases I did not prepare patches for:
1. The inclusion of the new public functions I wrote. You should test
them first and then you can incorporate them as you see fit.
2. There are some lines in your code that are long. As a rule of thumb,
long lines are a sign of complexity. Try to break up your code to
smaller functions, each doing the required work while together
contributed to something greater.
3. By convention, Lisp does not have closing parentheses on their own
line. Though I understand why this looks weird.
To ease your work with Emacs packaging, try something like this:
(use-package flymake
:ensure nil ; built into Emacs
:config
(setq flymake-fringe-indicator-position 'left-fringe)
(setq flymake-suppress-zero-counters t)
(setq flymake-start-on-flymake-mode t)
(setq flymake-no-changes-timeout nil)
(setq flymake-start-on-save-buffer t)
(setq flymake-proc-compilation-prevents-syntax-check t)
:bind (:map flymake-mode-map
("C-c ! s" . flymake-start)
("C-c ! d" . flymake-show-buffer-diagnostics) ; Emacs28
("C-c ! D" . flymake-show-project-diagnostics) ; Emacs28
("C-c ! n" . flymake-goto-next-error)
("C-c ! p" . flymake-goto-prev-error)))
(use-package package-lint-flymake
:ensure t
:config
(add-hook 'flymake-diagnostic-functions #'package-lint-flymake))
Go through the diagnostics buffer and address all the issues there.
--
Protesilaos Stavrou
https://protesilaos.com
Protesilaos Stavrou <info@protesilaos.com> writes:
>> From: relict007 <utils+sr.ht@kotlak.com>>> Date: Sun, 7 May 2023 20:14:59 +0530>>> [... 14 lines elided]>>>> I will check the details tomorrow. This is just an early comment. The>>> idea is to provide public functions. That way we cut down on>>> duplication and potential conflicts.>>>>>> More to follow.>>>> Sounds good.>> As promised, I made the requisite changes and pushed them to 'main'> branch.>> commit 694c1517be73949edbc3993c105c764da8e2571f> Author: Protesilaos Stavrou <info@protesilaos.com>> Date: Mon May 8 18:12:55 2023 +0300>> Expose public functions for returning links or backlinks>> We are doing the work in standalone functions that can be called from> Lisp, instead of limiting it to interactive functions. The code is> more composable as a result.>> This is in response to the mailing list thread started by relict007:> <https://lists.sr.ht/~protesilaos/denote/%3C87a5ygk6yi.fsf@kotlak.com%3E>>> relict007 is the developer of the 'denote-cache' package (in> progress): <https://git.sr.ht/~relict007/denote-cache>.>> README.org | 10 ++++++++++> denote.el | 46 +++++++++++++++++++++++++++++++---------------> 2 files changed, 41 insertions(+), 15 deletions(-)>> Please test the functions 'denote-link-return-links', 'denote-link-return-backlinks'.>>> [... 12 lines elided]>
Thanks for the changes Prot. I will test them and update back.
>>>> I have written a small package to cache Denote files' metadata in memory>>>> to help me with my specific use-cases.>>>>>>>> https://git.sr.ht/~relict007/denote-cache>>>>>> Congratulations! What is your plan with this? Maybe add it to some>>> package archive? We can include it in the Denote manual as well.>>>> Yes, I would like to add this to "gnu elpa" or otherwise to melpa. As>> this is my first Emacs package, I am not familiar with the listing>> process or their rules. I need to do some reading on that.>>>> It is fairly usable and I have been using it for some time now, however,>> before others start using it I would like to document it in detail>> including its configuration options, public functions, limitations,>> known issues, intended use cases etc. Also, as it doesn't provide any>> user facing functionality in itself, documentation needs few ready to>> use examples of its usage in various scenarios. Once it is at a>> satisfactory level, I will send a separate "new package announcement">> kind of mail to Denote mailing list. And I will be very happy if its get>> mentioned in the Denote manual.>> Okay, I can help you with the packaging. To put it on GNU ELPA you need> to assign copyright to the Free Software Foundation. This means that> you have to share your identity with them (though not with me or this> mailing list). MELPA has no such requirement.>
Ok, understood.
> I attach a few patches. They are not exhaustive, but should give you an> idea of how to tweak the code to meet the expected style.>> There are three cases I did not prepare patches for:>> 1. The inclusion of the new public functions I wrote. You should test> them first and then you can incorporate them as you see fit.>> 2. There are some lines in your code that are long. As a rule of thumb,> long lines are a sign of complexity. Try to break up your code to> smaller functions, each doing the required work while together> contributed to something greater.>> 3. By convention, Lisp does not have closing parentheses on their own> line. Though I understand why this looks weird.>> To ease your work with Emacs packaging, try something like this:>> (use-package flymake> :ensure nil ; built into Emacs> :config> (setq flymake-fringe-indicator-position 'left-fringe)> (setq flymake-suppress-zero-counters t)> (setq flymake-start-on-flymake-mode t)> (setq flymake-no-changes-timeout nil)> (setq flymake-start-on-save-buffer t)> (setq flymake-proc-compilation-prevents-syntax-check t)>> :bind (:map flymake-mode-map> ("C-c ! s" . flymake-start)> ("C-c ! d" . flymake-show-buffer-diagnostics) ; Emacs28> ("C-c ! D" . flymake-show-project-diagnostics) ; Emacs28> ("C-c ! n" . flymake-goto-next-error)> ("C-c ! p" . flymake-goto-prev-error)))>> (use-package package-lint-flymake> :ensure t> :config> (add-hook 'flymake-diagnostic-functions #'package-lint-flymake))>> Go through the diagnostics buffer and address all the issues there.
Many thanks for the patches and your generous guidance. I will go
through them and update "denote-cache" accordingly.
> From: relict007 <utils+sr.ht@kotlak.com>> Date: Tue, 9 May 2023 08:59:30 +0530> [... 121 lines elided]> Many thanks for the patches and your generous guidance. I will go> through them and update "denote-cache" accordingly.
Yes, sure. Note that my patches need not be installed and published. I
just thought this was the easiest way to annotate your file.
--
Protesilaos Stavrou
https://protesilaos.com
Hi Prot,
I pull'd the latest Denote and tested the newly added links
functions. With some adjustments to my code I was able to make it work,
however, there are a couple of points:
1. In function "denote-cache-rebuild-cache" we get a list of all Denote
files using "denote-directory-files" and try to get the metadata and
links of each file in a loop. However, the process pauses in between
when "denote-link-return-links" tries to open some large binary files
(e.g. with pdf Emacs asks if we want to open such a large file). Also,
it is kind of slow to open binary files with Emacs which causes initial
cache loading take lots of time. In my opinion if a Denote file is in
unknown format (i.e. its extension is not in "denote-file-types") we
should assume that there are no forward links in it and immediately
return without opening it at all.
2. The functions "denote-link-return-links" and
"denote-link-return-backlinks" raise a "user-error" in case no links are
found. I think it might be better to not raise an error as having no
links is totally normal and not exactly an error on user's
part. Functions should probably return 'nil' or an empty list.
This is my present code:
(defun denote-cache--retrieve-backlinks (file)
"Retrieve backlinks using denote native apis. No cache."
(condition-case err
(denote-link-return-backlinks file)
(user-error '())))
(defun denote-cache--retrieve-forwardlinks (file)
"Retrieve forward links, no cache."
;;TODO this is a temporary hack to speedup indexing, use denote
;;filetypes here
(if (equal (file-name-extension file) "org")
(condition-case err
(denote-link-return-links file)
(user-error '()))
'()))
thanks
P.S. I had already push'd your patches to denote-cache. Its fine as I
can see your comments directly in the code.
Protesilaos Stavrou <info@protesilaos.com> writes:
>> From: relict007 <utils+sr.ht@kotlak.com>>> Date: Tue, 9 May 2023 08:59:30 +0530>>> [... 121 lines elided]>>> Many thanks for the patches and your generous guidance. I will go>> through them and update "denote-cache" accordingly.>> Yes, sure. Note that my patches need not be installed and published. I> just thought this was the easiest way to annotate your file.
> From: relict007 <utils+sr.ht@kotlak.com>> Date: Tue, 9 May 2023 17:13:12 +0530>> Hi Prot,
Hello there!
> I pull'd the latest Denote and tested the newly added links> functions. With some adjustments to my code I was able to make it work,> however, there are a couple of points:>> 1. In function "denote-cache-rebuild-cache" we get a list of all Denote> files using "denote-directory-files" and try to get the metadata and> links of each file in a loop. However, the process pauses in between> when "denote-link-return-links" tries to open some large binary files> (e.g. with pdf Emacs asks if we want to open such a large file). Also,> it is kind of slow to open binary files with Emacs which causes initial> cache loading take lots of time. In my opinion if a Denote file is in> unknown format (i.e. its extension is not in "denote-file-types") we> should assume that there are no forward links in it and immediately> return without opening it at all.
Sure. Without looking at your code right now, I assume the solution is
to not use the 'denote-directory-files' but instead rely on
'denote-directory-text-only-files'.
> 2. The functions "denote-link-return-links" and> "denote-link-return-backlinks" raise a "user-error" in case no links are> found. I think it might be better to not raise an error as having no> links is totally normal and not exactly an error on user's> part. Functions should probably return 'nil' or an empty list.
Correct! I also got a message about this from Elias Storms and pushed
the changes a few minutes ago. The 'user-error' is limited to the
interactive functions.
> [... 23 lines elided]> P.S. I had already push'd your patches to denote-cache. Its fine as I> can see your comments directly in the code.
That's okay!
All the best,
Prot
--
Protesilaos Stavrou
https://protesilaos.com
Protesilaos Stavrou <info@protesilaos.com> writes:
>> From: relict007 <utils+sr.ht@kotlak.com>>> Date: Tue, 9 May 2023 17:13:12 +0530>>>> Hi Prot,>> Hello there!
Hi Prot,
Thanks for taking time to respond to my comments.
>>> I pull'd the latest Denote and tested the newly added links>> functions. With some adjustments to my code I was able to make it work,>> however, there are a couple of points:>>>> 1. In function "denote-cache-rebuild-cache" we get a list of all Denote>> files using "denote-directory-files" and try to get the metadata and>> links of each file in a loop. However, the process pauses in between>> when "denote-link-return-links" tries to open some large binary files>> (e.g. with pdf Emacs asks if we want to open such a large file). Also,>> it is kind of slow to open binary files with Emacs which causes initial>> cache loading take lots of time. In my opinion if a Denote file is in>> unknown format (i.e. its extension is not in "denote-file-types") we>> should assume that there are no forward links in it and immediately>> return without opening it at all.>> Sure. Without looking at your code right now, I assume the solution is> to not use the 'denote-directory-files' but instead rely on> 'denote-directory-text-only-files'.
Using 'denote-directory-text-only-files' to get a list of all files will
skip the binary files from denote-cache completely. What I was
suggesting earlier is that adding an early check for
'denote-file-has-supported-extension-p' in 'denote-link-return-links'
will allow us to skip a costly operation of opening the binary files.
In my function have added the check before I try to retrieve the forward
links, however, I feel it might be better if this check is in
'denote-link-return-links' itself. Please let me know what you think.
(defun denote-cache--retrieve-forwardlinks (file)
"Retrieve forward links, no cache."
(or (when (denote-file-has-supported-extension-p file)
(denote-link-return-forelinks file))
'()))
Unrelated to our present discussion, I think I have found an issue with
'denote-link-return-links'. This doesn't seem to be working for
denote links which have a newline in between. For example it doesn't
work when its like this:
[[denote:20220124T184558][Isaac
Asimov]]
However, this works:
[[denote:20220124T184558][Isaac Asimov]]
On hitting Enter, links open correctly in both cases. I have multiple
such links as I use 'fill-paragraph' frequently to align text. Let me
know if you can see this issue, otherwise I will setup some minimal
files to reproduce it.
>>> 2. The functions "denote-link-return-links" and>> "denote-link-return-backlinks" raise a "user-error" in case no links are>> found. I think it might be better to not raise an error as having no>> links is totally normal and not exactly an error on user's>> part. Functions should probably return 'nil' or an empty list.>> Correct! I also got a message about this from Elias Storms and pushed> the changes a few minutes ago. The 'user-error' is limited to the> interactive functions.
Thanks. I have updated my code accordingly.
>>> [... 23 lines elided]>>> P.S. I had already push'd your patches to denote-cache. Its fine as I>> can see your comments directly in the code.>> That's okay!>> All the best,> Prot
Thank You.
> From: relict007 <utils+sr.ht@kotlak.com>> Date: Sat, 13 May 2023 15:27:43 +0530> [... 12 lines elided]> Thanks for taking time to respond to my comments.
You are welcome!
> [... 15 lines elided]>> Sure. Without looking at your code right now, I assume the solution is>> to not use the 'denote-directory-files' but instead rely on>> 'denote-directory-text-only-files'.>> Using 'denote-directory-text-only-files' to get a list of all files will> skip the binary files from denote-cache completely. What I was> suggesting earlier is that adding an early check for> 'denote-file-has-supported-extension-p' in 'denote-link-return-links'> will allow us to skip a costly operation of opening the binary files.>> In my function have added the check before I try to retrieve the forward> links, however, I feel it might be better if this check is in> 'denote-link-return-links' itself. Please let me know what you think.
Sounds good! Do you want to prepare a patch?
> [... 7 lines elided]> Unrelated to our present discussion, I think I have found an issue with> 'denote-link-return-links'. This doesn't seem to be working for> denote links which have a newline in between. For example it doesn't> work when its like this:>>> [[denote:20220124T184558][Isaac> Asimov]]>> However, this works:>> [[denote:20220124T184558][Isaac Asimov]]>> On hitting Enter, links open correctly in both cases. I have multiple> such links as I use 'fill-paragraph' frequently to align text. Let me> know if you can see this issue, otherwise I will setup some minimal> files to reproduce it.
I don't have time to test it right now. I will do it tomorrow evening.
At first glance, perhaps 'denote-link--expand-identifiers' can benefit
from the approach of 'denote-link--link-at-point-string'? If yes, then
we need to consolidate the code to have one shared mechanism.
--
Protesilaos Stavrou
https://protesilaos.com
Protesilaos Stavrou <info@protesilaos.com> writes:
>> From: relict007 <utils+sr.ht@kotlak.com>>> Date: Sat, 13 May 2023 15:27:43 +0530>>> [... 12 lines elided]>>> Thanks for taking time to respond to my comments.>> You are welcome!>>> [... 15 lines elided]>>>> Sure. Without looking at your code right now, I assume the solution is>>> to not use the 'denote-directory-files' but instead rely on>>> 'denote-directory-text-only-files'.>>>> Using 'denote-directory-text-only-files' to get a list of all files will>> skip the binary files from denote-cache completely. What I was>> suggesting earlier is that adding an early check for>> 'denote-file-has-supported-extension-p' in 'denote-link-return-links'>> will allow us to skip a costly operation of opening the binary files.>>>> In my function have added the check before I try to retrieve the forward>> links, however, I feel it might be better if this check is in>> 'denote-link-return-links' itself. Please let me know what you think.>> Sounds good! Do you want to prepare a patch?
Hi Prot,
I have sent the patch in a separate email, not sure if it has reached to
you in a proper format. Let me know if you find some issues with it, or
with the code changes.
thanks
>>> [... 7 lines elided]>>> Unrelated to our present discussion, I think I have found an issue with>> 'denote-link-return-links'. This doesn't seem to be working for>> denote links which have a newline in between. For example it doesn't>> work when its like this:>>>>>> [[denote:20220124T184558][Isaac>> Asimov]]>>>> However, this works:>>>> [[denote:20220124T184558][Isaac Asimov]]>>>> On hitting Enter, links open correctly in both cases. I have multiple>> such links as I use 'fill-paragraph' frequently to align text. Let me>> know if you can see this issue, otherwise I will setup some minimal>> files to reproduce it.>> I don't have time to test it right now. I will do it tomorrow evening.> At first glance, perhaps 'denote-link--expand-identifiers' can benefit> from the approach of 'denote-link--link-at-point-string'? If yes, then> we need to consolidate the code to have one shared mechanism.