Hi fellow Denoters,
I have found an alternative for denote-get-path-by-id that
circumvents issue duplicate identifiers (such as exported
versions) by providing the user with options when this is the
case.
This solution lets me link to PDF, video and other files with
Denote identifiers, instead of only to txt, md and org. It also
serves as a warning that a duplicate file is available that might
need to be cleaned-up.
(defun denote-get-path-by-id (id)
"Return absolute path of ID string in `denote-directory-files'.
When more than one file with the same ID, use completion to select
target."
(let ((files (cl-remove-if-not
(lambda (f)
(string-prefix-p id (file-name-nondirectory f)))
(denote-directory-files))))
(if (> (length files) 1)
(completing-read "Select a file: " files)
(car files))))
Perhaps there is a more elegant way to express this idea, but it
works quite well in my workflow.
Regards
Peter
--
Dr Peter Prevos
---------------
peterprevos.com
> From: Peter Prevos <peter@prevos.net>> Date: Fri, 5 May 2023 19:42:31 +1000>> Hi fellow Denoters,
Hello Peter,
> I have found an alternative for denote-get-path-by-id that > circumvents issue duplicate identifiers (such as exported > versions) by providing the user with options when this is the > case.
I am in favour of a user option in this case. It seems we cannot have a
solution that "just works", so we need an option that informs users
about the specifics of the case. I would still prefer if Org had that
option, to the effect that exported files do not re-use the file name of
the original file. They could have a prefix, or a complete name that is
subject to a function that returns a string. But the Org source code is
not easy to work with, so I am not volunteering for such a task.
> This solution lets me link to PDF, video and other files with > Denote identifiers, instead of only to txt, md and org. It also > serves as a warning that a duplicate file is available that might > need to be cleaned-up.>> (defun denote-get-path-by-id (id)> "Return absolute path of ID string in `denote-directory-files'.> When more than one file with the same ID, use completion to select > target."> (let ((files (cl-remove-if-not> (lambda (f)> (string-prefix-p id (file-name-nondirectory f)))> (denote-directory-files))))> (if (> (length files) 1)> (completing-read "Select a file: " files)> (car files))))
Good idea! However, this assumes that the function will involve
interactive usage, which is not a given. The addition of interactivity
redefines the behaviour of the function. As such, it is better to think
of an approach that is complementary to what we already have, rather
than a direct substitute of it.
> Perhaps there is a more elegant way to express this idea, but it > works quite well in my workflow.
Yes, I think the code is fine as-is. Perhaps the 'completing-read' part
could be its own function, so that it can be tested individually, but
otherwise I am okay with this direction.
All the best,
Prot
--
Protesilaos Stavrou
https://protesilaos.com
Hi Prot, I did a fair bit of research on this issue so this is a
lengthy email
(plain text export from Org mode).
1 Summary of the issue
══════════════════════
• Denote can create and manage notes in three formats (Org,
Markdown
and plain text)
• The file naming convention can also be applied to other file
types,
such as other plain text formats, PDFs, images, video or audio
files
(attachments)
• Attachments can be linked to notes with
`denote-link-or-create'
• Auxiliary packages, such as "denote-menu" or "consult-notes
packages"
recognise Denote attachments (as listed in
`denote-directory-files')
• However, within Denote, only links to a recognised file type
can
be followed. Links can be created to attachments, but they
cannot be
followed.
This behaviour prevents issues with duplicate identifiers, which
can
occur when exporting Denote files. In most cases, exported files
are
stored in the same directory as the source and files have the
same
name sans extension,
e.g. `20230507T152924--denote-file-linking-issue__emacs_todo.org'
and
`20230507T152924--denote-file-linking-issue__emacs_todo.pdf'.
The `denote-file-is-note-p' clause in the
`denote-get-path-by-id'
function only allows links to notes to be followed. This
behaviour is
undesired. Being able to add links to Denote attachments, but
not
being able to follow them is contradictory. The current
behaviour
prevents users from fully utilising the Denote file-naming
convention. Denote should honour attachments as there is scope
to
develop extensions to manage collections of photos, videos,
scanned
documents and so on.
2 Solutions
═══════════
There are two solution spaces to solve this problem:
1. Modify Denote to manage the potential conflicts with
duplicate
identifiers
2. Prevent duplicate identifiers from being created in the first
place
2.1 Denote
──────────
The `denote-get-path-by-id' can be modified to ignore duplicate
filenames, rather than ignoring attachments. The version below
fetches
the relevant Denote file when an exported version(s) exist:
┌────
│ (defun denote-get-path-by-id (id)
│ "Return absolute path of ID string in
`denote-directory-files'.
│ When duplicate Denote IDs exist, ignore attachments."
│ ;; Find all files with the sought Denote ID
│ (if-let ((files (cl-remove-if-not
│ (lambda (f)
│ (string-prefix-p id (file-name-nondirectory
f)))
│ (denote-directory-files))))
│ (if (> (length files) 1)
│ ;; If there are duplicates, only use the first note
│ (car (cl-mapcan (lambda (file denote-val)
│ (when denote-val
│ (list file)))
│ files
│ (mapcar 'denote-file-is-note-p files)))
│ ;; Else, use the value that was found
│ (car files))
│ (user-error "Denote identifier not found")))
└────
There is one scenario where this solution does not work
perfectly,
which is the rare case when the user exports a note from Org to
plain
text. In cases where more than one version with a valid Denote
extension exists, the Org Mode file will be selected as it will
be the
first in the list.
This code might not be the most efficient method (partially
GPT-generated), but it does the trick and is fully tested.
2.2 Prevent duplicate identifiers due to exports
────────────────────────────────────────────────
Org and Markdown use different approaches to exporting files. No
solution is required for plain text files as there is no
standardised
export functionality.
2.2.1 Org Mode
╌╌╌╌╌╌╌╌╌╌╌╌╌╌
Org has a built-in export engine. Org exports are configurable
to
prevent duplicate identifiers when exporting, this can either be
achieved manually for each exported file, or by advising the Org
export functions.
◊ 2.2.1.1 Manual
User should insert `#+export_file_name: <filename>' in front
matter
before exporting to force a filename.
Disadvantage of this method is that it depends on user behaviour
and
non-compliance leads to duplicate identifiers.
◊ 2.2.1.2 Automated
Org export functionality can be customised with hooks or by
advising
functions. This solution moves all exports to another folder
(http://rwx.io/posts/org-export-configurations/):
┌────
│ (defvar org-export-output-directory-prefix "~/export_" "prefix
of directory used for org-mode export")
│
│ (defadvice org-export-output-file-name (before
org-add-export-dirr activate)
│ "Modifies org-export to place exported files in a
different directory"
│ (when (not pub-dir)
│ (setq pub-dir (concat org-export-output-directory-prefix
(substring extension 1)))
│ (when (not (file-directory-p pub-dir))
│ (make-directory pub-dir))))
└────
The export directory should not be a child of the Denote
directory, or
it should be excluded with the
`denote-excluded-directories-regexp'
variable.
Note that using a different directory will require some
additional
configuration when exporting using LaTeX. The export folder
cannot be inside the path of the `denote-directory' to prevent
Denote
from recognising it as an attachment.
https://emacs.stackexchange.com/questions/45751/org-export-to-different-directory
This approach can also be used to configure the name of exported
files. This function asks the user to enter a filename when the
source
file is a Denote note and no export file name is specified in
the
front matter.
┌────
│ (defun my-org-export-extract-export-file-name ()
│ "Extract EXPORT_FILE_NAME from Org buffer."
│ ;; Copied from org-export-output-file-name source code
│ (org-with-point-at (point-min)
│ (catch :found
│ (let ((case-fold-search t))
│ (while (re-search-forward
│ "^[ \t]*#\\+EXPORT_FILE_NAME:[ \t]+\\S-" nil t)
│ (let ((element (org-element-at-point)))
│ (when (eq 'keyword (org-element-type element))
│ (throw :found
│ (org-element-property :value element)))))))))
│
│ (defun my-org-export-output-file-name-advice (extension
&optional subtreep pub-dir)
│ ;; Check if the file is a Denote note and whether
expor_file_name exists
│ (when (and (denote-file-is-note-p buffer-file-name)
│ (not (my-org-export-extract-export-file-name)))
│ ;; Enter new file name (sans extension)
│ (let ((filename (read-file-name "Export file name (without
extension): ")))
│ (concat (expand-file-name filename) (princ
extension)))))
│
│ (advice-add 'org-export-output-file-name
│ :before-until
│ #'my-org-export-output-file-name-advice)
└────
◊ 2.2.1.3 Publishing
Org Mode also has a publishing tool for exporting a collection
of
files. Some user might apply this approach to convert their note
collection to a public or private website.
The `org-publish-project-alist' variable drives the publishing
process, including the publishing directory.
The publishing directory should not be a child of the Denote
directory, or it should be excluded with the
`denote-excluded-directories-regexp' variable.
2.2.2 Markdown
╌╌╌╌╌╌╌╌╌╌╌╌╌╌
Exporting from Markdown requires an external processor (e.g.,
Markdown.pl, Pandoc, or MultiMarkdown). The `markdown-command'
variable defines the command line used in export, fo example:
`(setq
markdown-command "multimarkdown")'.
The export process thus occurs outside of Emacs and cannot be
managed
with configuration. Users need to read the documentation of
their
preferred processor to prevent duplicate filename identifiers.
3 Conclusion
════════════
The current version of Denote does not follow links to
attachments. This behaviour is contradictory with other
functions and
auxiliary packages that recognise attachments as being part of
the
note collection.
Recommendations:
1. Improve the `denote-get-path-by-id' function to allow linking
to
valid attachments (ignoring exported versions of notes).
2. Add a section to the Denote manual about exporting notes with
examples on how to prevent duplicate identifiers.
Apologies for the lengthy response.
Regards
P:)
Hi Peter,
> The `denote-get-path-by-id' can be modified to ignore duplicate> filenames, rather than ignoring attachments.
What does your attachment file name look like? How do you differentiate attachment to another note and duplicates?
nobiot
Sent from my iPhone
Hi Nobiot,
An attachment can be anything, they are not generated with Denote
but manually. For example, I could take a photo today and call it:
20230514T175032--kangaroos__nature.jpg or I scan attachments for a
meeting and call them
20230430T172742--master-class-induction__magic.pdf, video files,
audio recordings etc etc. An attachment can be anything that I
like to keep within the Denote ecosystem. There are also text
files, such as R and Fountain scripts, that I save using this
format.
I save these in subfolders of my denote-directory, but that is not
necessary.
This way Denote becomes more than a note-taking package but also
an archivist for anything that relates to my notes.
Not being able to follow links to attachments is a major issue for
me.
Regards
P:)
Noboru <me@nobiot.com> writes:
> Hi Peter,>>> The `denote-get-path-by-id' can be modified to ignore duplicate>> filenames, rather than ignoring attachments. >>> What does your attachment file name look like? How do you > differentiate attachment to another note and duplicates?>> nobiot > Sent from my iPhone
--
Dr Peter Prevos
---------------
peterprevos.com
> An attachment can be anything, they are not generated with Denote but manually. For example, I could take a photo today and call it: 20230514T175032--kangaroos__nature.jpg or I scan attachments for a meeting and call them 20230430T172742--master-class-induction__magic.pdf, video files, audio recordings etc etc. An attachment can be anything that I like to keep within the Denote ecosystem. There are also text files, such as R and Fountain scripts, that I save using this format. > I save these in subfolders of my denote-directory, but that is not necessary.
Okay, thank you for this. So they are files with an denote-compliant ID in the file name, with an extension that is not denote-type.
> This way Denote becomes more than a note-taking package but also an archivist for anything that relates to my notes. > Not being able to follow links to attachments is a major issue for me.
Somehow I do not seem to have the same issue on my end. I may well be mistaken. I am away from my PC this weekend; I will double-check this when I return to it and come back to the thread.
Hi Peter,
Even I don't see this issue happening in my setup. Could you please
recheck if you are on the latest Denote, i.e. the latest git version
(and not the one in GNU ELPA)
I had a similar issue sometimes back and it was fixed here:
https://github.com/protesilaos/denote/issues/135
Also, @nobiot fixed a related issue here:
https://github.com/protesilaos/denote/issues/142
thanks
Noboru <me@nobiot.com> writes:
>> An attachment can be anything, they are not generated with Denote>> but manually. For example, I could take a photo today and call it:>> 20230514T175032--kangaroos__nature.jpg or I scan attachments for a>> meeting and call them>> 20230430T172742--master-class-induction__magic.pdf, video files,>> audio recordings etc etc. An attachment can be anything that I like>> to keep within the Denote ecosystem. There are also text files, such>> as R and Fountain scripts, that I save using this format.>> I save these in subfolders of my denote-directory, but that is not necessary.>> Okay, thank you for this. So they are files with an denote-compliant ID in the file name, with an extension that is not denote-type.>>> This way Denote becomes more than a note-taking package but also an archivist for anything that relates to my notes. >> Not being able to follow links to attachments is a major issue for me.>> Somehow I do not seem to have the same issue on my end. I may well be mistaken. I am away from my PC this weekend; I will double-check this when I return to it and come back to the thread.
Hi Nobiot and relict007,
I am using the Melpa version and just checked the GitHub
version. Apparent I wasted my time worrying ... :)
Please ignore my email.
Perhaps the text and code I wrote about exporting would be good to
add to the manual to prevent the duplicate ID issue in the first
place.
Apologies to waste your time.
P:)
> From: Peter Prevos <peter@prevos.net>> Date: Sun, 14 May 2023 19:11:49 +1000>> Hi Nobiot and relict007,
Hello folks, hello Peter,
> [... 9 lines elided]> Apologies to waste your time.
No worries! I also got the impression this was not fixed. All good
now.
* * *
Just to note that I also use Denote for all sorts of files. Some are
subdirectories of 'denote-directory', while others are on their own
(e.g. videos).
All the best,
Prot
--
Protesilaos Stavrou
https://protesilaos.com
> From: Peter Prevos <peter@prevos.net>> Date: Sun, 14 May 2023 09:12:15 +1000>> Hi Prot, I did a fair bit of research on this issue so this is a > lengthy email> (plain text export from Org mode).> [... 265 lines elided]
Hello Peter,
In your last email in this thread you mentioned that while the linking
has been solved, we could still include some of your code in the manual.
Can you exctract it from here and send it on its own so that we can
check and discuss it?
All the best,
Prot
--
Protesilaos Stavrou
https://protesilaos.com
Hi Prot,
Perhaps a section in the Denote manual covering exporting Org and
Markdown files would be useful.
You can use this as you see fit:
Exporting Denote notes
──────────────────────
When exporting Denote notes to, for example, a website or PDF,
there is a risk that the same file name is used with a new
extension. This can be problematic because it creates files with
duplicate identifiers, e.g. 20230515T085612--example.org and
20230515T085612--example.pdf.
Org Mode and Markdown use different approaches to exporting files.
No solution is required for plain text files as there is no
standardised export functionality for this format.
Org Mode
╌╌╌╌╌╌╌╌
Org Mode has a built-in configurable export engine. You can
prevent duplicate identifiers when exporting manually for
each exported file or by advising the Org export function.
◊ Manual configuration
Insert `#+export_file_name: <filename>' in front the matter
before exporting to force a filename.
The disadvantage of this method is that it depends on the
user's behaviour. Forgetting to add a new name can lead to
duplicate identifiers.
◊ Store exports in another folder
This solution moves all exports to another folder
(http://rwx.io/posts/org-export-configurations/):
(defvar org-export-output-directory-prefix "../export_" "prefix
of directory used for org-mode export")
(defadvice org-export-output-file-name (before
org-add-export-dirr activate)
"Modifies org-export to place exported files in a
different directory"
(when (not pub-dir)
(setq pub-dir (concat org-export-output-directory-prefix
(substring extension 1)))
(when (not (file-directory-p pub-dir))
(make-directory pub-dir))))
Please note that the export directory should not be a child of
the Denote directory or it should be excluded with the
`denote-excluded-directories-regexp' variable.
Using a different directory will require some
additional configuration when exporting using LaTeX. The export
folder cannot be inside the path of the `denote-directory' to
prevent Denote from recognising it as an attachment.
https://emacs.stackexchange.com/questions/45751/org-export-to-different-directory
◊ Ask for new name when not defined
This function asks the user to enter a filename when the
source file is a Denote note, and no export file name is specified in
the front matter.
(defun my-org-export-extract-export-file-name ()
"Extract EXPORT_FILE_NAME from Org buffer."
;; Copied from org-export-output-file-name source code
(org-with-point-at (point-min)
(catch :found
(let ((case-fold-search t))
(while (re-search-forward
"^[ \t]*#\\+EXPORT_FILE_NAME:[ \t]+\\S-" nil t)
(let ((element (org-element-at-point)))
(when (eq 'keyword (org-element-type element))
(throw :found
(org-element-property :value element)))))))))
(defun my-org-export-output-file-name-advice (extension
&optional subtreep pub-dir)
;; Check if the file is a Denote note and whether
expor_file_name exists
(when (and (denote-file-is-note-p buffer-file-name)
(not (my-org-export-extract-export-file-name)))
;; Enter new file name (sans extension)
(let ((filename (read-file-name "Export file name (without
extension): ")))
(concat (expand-file-name filename) (princ
extension)))))
(advice-add 'org-export-output-file-name
:before-until
#'my-org-export-output-file-name-advice)
◊ Org Mode Publishing
Org Mode also has a publishing tool for exporting a collection
of
files. Some user might apply this approach to convert their note
collection to a public or private website.
The `org-publish-project-alist' variable drives the publishing
process, including the publishing directory.
The publishing directory should not be a child of the Denote
directory, or it should be excluded with the
`denote-excluded-directories-regexp' variable.
2.2.2 Markdown
╌╌╌╌╌╌╌╌╌╌╌╌╌╌
Exporting from Markdown requires an external processor (e.g.,
Markdown.pl, Pandoc, or MultiMarkdown). The `markdown-command'
variable defines the command line used in export, fo example:
`(setq
markdown-command "multimarkdown")'.
The export process thus occurs outside of Emacs and cannot be
managed
with configuration. Users need to read the documentation of
their
preferred processor to prevent duplicate filename identifiers.
Regard
Peter
> From: Peter Prevos <peter@prevos.net>> Date: Mon, 15 May 2023 09:04:27 +1000>> Hi Prot,>> Perhaps a section in the Denote manual covering exporting Org and > Markdown files would be useful.>> You can use this as you see fit:> [... 132 lines elided]
Hello Peter,
Thank you for doing this! I just made all the changes. I used your
text as a basis. I rewrote the advice that adds the exported file to a
directory. I also omitted the second advice with the prompt, because
having two many advices will be a problem for users who are not sure
what they are doing.
All the best,
Prot
--
Protesilaos Stavrou
https://protesilaos.com