[ Sorry, re-sending this as the attached Org files are not accepted by
the server. I add their contents below. ]
> From: Protesilaos Stavrou <info@protesilaos.com>> Date: Tue, 26 Jul 2022 12:57:54 +0300>>> From: Peter Prevos <peter@prevos.net>>> Date: Tue, 26 Jul 2022 19:26:27 +1000>>>> Hi Prot and Hanspeter,>> Hello Peter, Hanspeter,>>> I agree there is no need from a code perspective to have symmetry >> between file name and the front matter. The power of Denote is its >> reliance on a file naming convention.>>>> However, when renaming a file, the Denote does seem to retrieve data >> from the front matter. The 'denote-retrieve--*-front-matter-key-regexp' >> variables seem to look at the file content - assuming I read the source >> correctly.>> You read the source correctly. We use this functionality for> non-critical purposes. This is what I had in mind when I wrote about> "convenience".>> To be clear though: I am not downplaying the value this has and I want> us to keep this feature and improve on it wherever we can. Just> clarifying that Denote will work even without it.>>> Synchronising front matter and file name would be good data integrity >> practice and provide some convenience. It means that you only have to >> change the front matter and run the synchronise function.>>>> The file name helps with heuristics, the front matter improves >> readability. For Org mode files it also means that you can use its >> functionality ('org-element-parse-buffer') to extract metadata about the >> file.>> Indeed. This is where the front matter shows its value. It is good for> usability and makes it possible to leverage non-Denote functionality.>>> I crafted a function to create a file name from Org mode front matter:>>>>> [... 16 lines elided]>>> I can't get the tags right. Note that the canonical format for Org mode >> filetags is ":tag1:tag2" instead of "tag1 tag2". It would be nice if >> Denote can also follow this syntax. The Org function (org-split-string) >> takes care of tags.>> We discussed this on the mailing list before.[1] The finding was that> what we currently use is---or seems to be---technically correct. Have> you encountered cases where it returns faulty values?>> I attach two files which differ in their filetags syntax. Both yield> identical results on my end.>> [1] https://lists.sr.ht/~protesilaos/denote/%3C875yk1x8en.fsf@m4x.org%3E#%3C87mtd8m82n.fsf@m4x.org%3E>>> Markdown mode has similar functionality to extract YAML or TOML >> metadata.>>>> Synchronising front matter and file name would function as a text-only >> replacement for a database and allow for extensibility with Org and >> Markdown modes. One could, for example add '#+subtitle:' or #+status:' >> etc etc. There is only so much info you can shove into a file name ...>> I agree. The key now is to think about the particularities. What do we> want our function to do, how do we optimise it at scale, and the like.>> All the best,> Prot20220726T123815--this-is-a-test__denote_testing.org
#+title: This is a test
#+date: [2022-07-26 Tue 12:38]
#+filetags: denote testing
#+identifier: 20220726T123815
* This is a heading which inherits the filetags
#+begin_src emacs-lisp
(org-get-tags)
#+end_src
#+RESULTS:
| denote | testing |
20220726T123915--this-is-a-second-test__denote_testing.org
#+title: This is a second test
#+date: [2022-07-26 Tue 12:38]
#+filetags: :denote:testing
#+identifier: 20220726T123815
* This is a heading which inherits the filetags
#+begin_src emacs-lisp
(org-get-tags)
#+end_src
#+RESULTS:
| denote | testing |
--
Protesilaos Stavrou
https://protesilaos.com
Hello Prot and Peter
Very interesting discussion; thank you both very much.
The following is a somewhat lengthy write-down (sorry for that)
of my thoughts, based on what you have written previously.
I guess it boils down to the following question:
Does a user want to keep her (.org, .md, and possibly other supported)
file's frontmatter in sync with the filename - or not.
a) if not: denote-dired-rename-file does exactly this, no?
b) if yes: denote-dired-rename-file-and-add-front-matter is not really sufficient, yes?
I understand:
- that having related informaiton (with same key) at two places (filename and
frontmatter) is problematic - if the user expects the information to
be in-sync.
- that having the frontmatter is very convenient for multiple purposes:
export, print, grep-ing the files content, etc.
The question is perhaps:
What is the user's 'natural' expectation?
a) the frontmatter may be totally out-of-sync to the filename
b) the two are in-sync.
I personally did / would expect / want the two to be in-sync;
because at one time I want to find a file based e.g. on its keywords by means
of dired, another time I want to find it via grep.
Of course, trying to sync filename and frontmatter somehow poses the
usual problems as what to do if there is a conflict between the two or
which one 'wins' if they are out-of-sync.
Perhaps it would be best to let the user decide?
(note: when I speak of 'syncing', then I mean, that only those (org)
properties '#+..:' are touched, which have their counterpart in the
filename; i.e. title, keywords and (perhaps, see below) identifier / date.
Any other property shall not be touched.
What do you think of the following:
A 'rename' function would either:
- just change the filename
(either on users request or anyways, if the file's filetype
is unsupported by denotes frontmatter handling), or
- sync all information contained in the filename (id (date/time), title,
keywords) to the frontmatter; if according properties do not exist,
they are added at (e.g.) at the front, if they exist, they are kept
at existing location but their values are replaced.
(note: any other file's existing property remains unchanged)
If more than one keyword with same key exists in the file
(e.g. multiple #+keyword lines, then all but one are deleted).
A 'reverse-rename' function would:
- replace (rename) those parts of the filename, which have an existing
correspondence in the frontmatter, i.e. title, keywords, date/time, identifier
(if e.g. title does not exist as frontmatter, the filename's title
remains unchanged, if it exists, the filename's title is updated
accordingly).
- there remains the problem, however, that #+idnetifier and #+date might
not be in-sync themselves; a clear rule of preference would perhaps be a solution?
- it is even questionable, whether a filename's identifier should ever
be changed / overwritten by such a reverse-rename operation, since
in-bound links to this file then would break, no?
- because of this, perhaps it would even make sense to define,
that the frontmatter's #+identifier is always synched from
filename-to-frontmatter, never in reverse direction.
- Then the question arises: what is the 'sematics' of the #+date
property in the frontmatter:
a) is it just another (more human readable) way of the same thing
as #+identifier, or
b) is it meant to be some arbitrary date (having some arbitrary
meaning to the author of the note, i.e. not necessarily meaning
creation date of the note)?
Depending on the answer (and semantic definition),
#+date would be synced or not..
Ok, sorry again for the very lengthy musings of mine.
However, it interests me: what do you think:
- would this be usable / useful?
- what are the pitfalls?
- are there edge cases, where this makes no sense?
- did I even get the basic ideas / assumptions of Denote right?
- etc.
Best Regards to both of you
Hanspeter
Protesilaos Stavrou <info@protesilaos.com> writes:
> [ Sorry, re-sending this as the attached Org files are not accepted by> the server. I add their contents below. ]>>> From: Protesilaos Stavrou <info@protesilaos.com>>> Date: Tue, 26 Jul 2022 12:57:54 +0300>>>>> From: Peter Prevos <peter@prevos.net>>>> Date: Tue, 26 Jul 2022 19:26:27 +1000>>>>>> Hi Prot and Hanspeter,>>>> Hello Peter, Hanspeter,>>>>> I agree there is no need from a code perspective to have symmetry >>> between file name and the front matter. The power of Denote is its >>> reliance on a file naming convention.>>>>>> However, when renaming a file, the Denote does seem to retrieve data >>> from the front matter. The 'denote-retrieve--*-front-matter-key-regexp' >>> variables seem to look at the file content - assuming I read the source >>> correctly.>>>> You read the source correctly. We use this functionality for>> non-critical purposes. This is what I had in mind when I wrote about>> "convenience".>>>> To be clear though: I am not downplaying the value this has and I want>> us to keep this feature and improve on it wherever we can. Just>> clarifying that Denote will work even without it.>>>>> Synchronising front matter and file name would be good data integrity >>> practice and provide some convenience. It means that you only have to >>> change the front matter and run the synchronise function.>>>>>> The file name helps with heuristics, the front matter improves >>> readability. For Org mode files it also means that you can use its >>> functionality ('org-element-parse-buffer') to extract metadata about the >>> file.>>>> Indeed. This is where the front matter shows its value. It is good for>> usability and makes it possible to leverage non-Denote functionality.>>>>> I crafted a function to create a file name from Org mode front matter:>>>>>>>> [... 16 lines elided]>>>>> I can't get the tags right. Note that the canonical format for Org mode >>> filetags is ":tag1:tag2" instead of "tag1 tag2". It would be nice if >>> Denote can also follow this syntax. The Org function (org-split-string) >>> takes care of tags.>>>> We discussed this on the mailing list before.[1] The finding was that>> what we currently use is---or seems to be---technically correct. Have>> you encountered cases where it returns faulty values?>>>> I attach two files which differ in their filetags syntax. Both yield>> identical results on my end.>>>> [1] https://lists.sr.ht/~protesilaos/denote/%3C875yk1x8en.fsf@m4x.org%3E#%3C87mtd8m82n.fsf@m4x.org%3E>>>>> Markdown mode has similar functionality to extract YAML or TOML >>> metadata.>>>>>> Synchronising front matter and file name would function as a text-only >>> replacement for a database and allow for extensibility with Org and >>> Markdown modes. One could, for example add '#+subtitle:' or #+status:' >>> etc etc. There is only so much info you can shove into a file name ...>>>> I agree. The key now is to think about the particularities. What do we>> want our function to do, how do we optimise it at scale, and the like.>>>> All the best,>> Prot>> 20220726T123815--this-is-a-test__denote_testing.org>> #+title: This is a test> #+date: [2022-07-26 Tue 12:38]> #+filetags: denote testing> #+identifier: 20220726T123815>> * This is a heading which inherits the filetags>> #+begin_src emacs-lisp> (org-get-tags)> #+end_src>> #+RESULTS:> | denote | testing |>>>> 20220726T123915--this-is-a-second-test__denote_testing.org>> #+title: This is a second test> #+date: [2022-07-26 Tue 12:38]> #+filetags: :denote:testing> #+identifier: 20220726T123815>> * This is a heading which inherits the filetags>> #+begin_src emacs-lisp> (org-get-tags)> #+end_src>> #+RESULTS:> | denote | testing |>>> -- > Protesilaos Stavrou> https://protesilaos.com
> From: Hanspeter Gisler <hpgisleropen@bluewin.ch>> Date: Tue, 26 Jul 2022 22:53:21 +0200>> Hello Prot and Peter
Hello Hanspeter, Peter,
> Very interesting discussion; thank you both very much.
You are welcome!
> The following is a somewhat lengthy write-down (sorry for that)> of my thoughts, based on what you have written previously.
No worries! I think it is better to have clearly elucidated views on
the matter. It is important to understand what the needs are and how
best to address them.
> I guess it boils down to the following question:>> Does a user want to keep her (.org, .md, and possibly other supported)> file's frontmatter in sync with the filename - or not.
I think we all agree that syncing is preferable. It is better for the
user for the reasons you also mentioned (grep, print, export, etc.).
> a) if not: denote-dired-rename-file does exactly this, no?
It is more nuanced. This command updates the front matter if it already
exists. If there is no front matter, it simply renames the file. The
idea with this is that it can operate on any file type, including
writable ones. When it operates on a file with front matter, which we
can count as a "note" from the user's perspective, it does the job of
syncing.
[ Whether it does it well enough is another discussion. ]
> b) if yes: denote-dired-rename-file-and-add-front-matter is not really sufficient, yes?
You are right: it isn't sufficient for all scenaria. It simply adds the
front matter as a whole block. It does not check if only one line is
missing and, generally, does not do anything smart. It simply goes to
the top of the buffer and inserts while evaluating the appropriate
variable, such as 'denote-org-front-matter'.
> I understand:>> - that having related informaiton (with same key) at two places (filename and> frontmatter) is problematic - if the user expects the information to> be in-sync.
It is problematic for the user and we want to help them avoid such a
case.
> - that having the frontmatter is very convenient for multiple purposes:> export, print, grep-ing the files content, etc.
Agreed!
> The question is perhaps:> What is the user's 'natural' expectation?>> a) the frontmatter may be totally out-of-sync to the filename> b) the two are in-sync.
Point b.
> I personally did / would expect / want the two to be in-sync;> because at one time I want to find a file based e.g. on its keywords by means> of dired, another time I want to find it via grep.
Having commented on the aforementioned renaming commands, I must ask:
have they not worked at all for you?
> Of course, trying to sync filename and frontmatter somehow poses the> usual problems as what to do if there is a conflict between the two or> which one 'wins' if they are out-of-sync.> Perhaps it would be best to let the user decide?
We can answer this question at a latter point. I think once we have all
the tools in place, it will be easier to assess the utility of a user
option. Introducing it too early may put us in a precarious spot.
> (note: when I speak of 'syncing', then I mean, that only those (org)> properties '#+..:' are touched, which have their counterpart in the> filename; i.e. title, keywords and (perhaps, see below) identifier / date.> Any other property shall not be touched.
Agreed!
> What do you think of the following:>> A 'rename' function would either:>> - just change the filename> (either on users request or anyways, if the file's filetype> is unsupported by denotes frontmatter handling), or
I am not sure I follow the parenthetic statement here. That granted, I
will try to comment: the 'denote-dired-rename-file' renames any file and
only rewrites the front matter (subject to a "yes-or-no" prompt) if it
finds one.
> - sync all information contained in the filename (id (date/time), title,> keywords) to the frontmatter; if according properties do not exist,> they are added at (e.g.) at the front, if they exist, they are kept> at existing location but their values are replaced.> (note: any other file's existing property remains unchanged)> If more than one keyword with same key exists in the file> (e.g. multiple #+keyword lines, then all but one are deleted).
Note that the identifier should never be modified in the file name and,
by extension, should remain intact if shown in the front matter.
(Remember that Denote's linking only needs the file name, not the front
matter.) Changing the identifier is a recipe for breakage: links would
stop working and we could easily end up with lots of duplicates.
I guess the idea of changing the identifier is to reflect a new date? I
think there are other ways to do that, such as by relying on file
attributes or, a bit more advanced setups, a Git (VCS in general)
history. For example, 'ls' accepts a '--sort' flag which can apply to
the time: <https://protesilaos.com/emacs/denote#h:a7fd5e0a-78f7-434e-aa2e-e150479c16e2>
That aside, I agree with what you write there. We want to add what is
missing, replace old values, and delete duplicate entries.
> A 'reverse-rename' function would:>> - replace (rename) those parts of the filename, which have an existing> correspondence in the frontmatter, i.e. title, keywords, date/time, identifier> (if e.g. title does not exist as frontmatter, the filename's title> remains unchanged, if it exists, the filename's title is updated> accordingly).
I like this idea! Though it should not touch the identifier, as
explained above.
Of course, we cannot prevent the user from editing whatever they want.
One can, for instance, rename the file with the standard renaming
facilities or wdired and edit the identifier. Any resulting errors
would not be our fault though. It is important for our purposes not to
cause breakage, as that jeopardises the trust users put in the package.
> - there remains the problem, however, that #+idnetifier and #+date might> not be in-sync themselves; a clear rule of preference would perhaps be a solution?
Why is it a problem? The identifier is derived from the date because it
feels more natural to have, say, 20220727T060925 than apparent gibberish
like v7fd5e0a-78f7-434e-aa2e-a150479c16e9. The identifier is for
identification, but it remains sufficiently readable for practical
purposes.
The #+date can be whatever the user wants. Perhaps we could even
provide or document a convenience function which appends a
#+lastmodified (or however we may call it) upon save? I am interested
to learn more about this and I am not against the idea.
In my workflow, if I need to find files by date of modification, I
update Dired's 'ls' flags (check the link to the manual). If I need to
review the history of a file, I rely on Git. For example, the command
'vc-print-log' (bound to 'C-x v l' by default), produces the log of
commits pertinent to the file. Typing 'd' in that buffer generates a
diff for the commit at point.
> - it is even questionable, whether a filename's identifier should ever> be changed / overwritten by such a reverse-rename operation, since> in-bound links to this file then would break, no?
Correct! We don't want to break things.
> - because of this, perhaps it would even make sense to define,> that the frontmatter's #+identifier is always synched from> filename-to-frontmatter, never in reverse direction.
Yes, we could arrange for this.
> - Then the question arises: what is the 'sematics' of the #+date> property in the frontmatter:> a) is it just another (more human readable) way of the same thing> as #+identifier, or> b) is it meant to be some arbitrary date (having some arbitrary> meaning to the author of the note, i.e. not necessarily meaning> creation date of the note)?> Depending on the answer (and semantic definition),> #+date would be synced or not..
As with all front matter, the #+date is for the user's convenience. It
reflects the time the note was created. It is not used for
identification purposes and performs no critical function in Denote's
code. Perhaps the user expects the #+date to show the last modification
time or something else? I am happy to support that use-case, provided
we understand what the needs are.
> Ok, sorry again for the very lengthy musings of mine.> However, it interests me: what do you think:
No worries about the length and thanks for taking the time to write
this! It helps us understand how things stand.
> - would this be usable / useful?
Syncing is useful and desirable. I want it and, I think, every user
wants it.
> - what are the pitfalls?
Front matter is for the user's convenience. Denote does not perform any
critical function that is contingent on the front matter. Links and
backlinks read the identifier from the file name.
As I have already implied, "user's convenience" does not make it
unimportant. We still want to take good care of it.
> - are there edge cases, where this makes no sense?
I think for as long as the file satisfied 'denote--only-note-p' we have
nothing to worry about. We are anyway prompting for confirmation before
operating on file contents and we do not save the buffer after the fact.
For as long as we remain cautious, we have no reason to worry about edge
cases. All we need now is to start writing the code we need to improve
on what we have.
> - did I even get the basic ideas / assumptions of Denote right?> - etc.
I think you did. My reply is verbose in the interest of the reader who
is not necessarily aware of the minutia.
All the best,
Prot
--
Protesilaos Stavrou
https://protesilaos.com
Protesilaos Stavrou <info@protesilaos.com> writes:
>> From: Hanspeter Gisler <hpgisleropen@bluewin.ch>>> Date: Tue, 26 Jul 2022 22:53:21 +0200> Hello Hanspeter, Peter,
[...]
Hello Prot and Peter
[...]
>> b) the two are in-sync.>> Point b.>>> I personally did / would expect / want the two to be in-sync;>> because at one time I want to find a file based e.g. on its keywords by means>> of dired, another time I want to find it via grep.>> Having commented on the aforementioned renaming commands, I must ask:> have they not worked at all for you?
In my case, was confronted with the following situation:
I had a bunch of .org files which I had created before starting to use
Denote and which I wanted to now be handled by Denote.
They all had the following frontmatter:
#+title: ...
#+date: ...
#+filetags: ...
But of course no:
#+identifier:
as this is Dentoe specific, I believe.
So the 'denote-dired-rename-file' commnd did update correctly the first
3 properties, but did not add the #+identifier: property.
I am aware, that this is an edge case and this problem would not have
existed if I had created the original notes with Denote in the first
place. So once edited manually, the problem goes away.
So, perhaps it is not justified to put a lot of effort into handling
this specific use-case...?
[...]
>>> What do you think of the following:>>>> A 'rename' function would either:>>>> - just change the filename>> (either on users request or anyways, if the file's filetype>> is unsupported by denotes frontmatter handling), or>> I am not sure I follow the parenthetic statement here. That granted, I> will try to comment: the 'denote-dired-rename-file' renames any file and> only rewrites the front matter (subject to a "yes-or-no" prompt) if it> finds one.
By this I meant:
If the rename function has the following functionalities
(similar to the currently existing 'denote-dired-rename-file'):
- detect, whether the to be renamed file is of
denote-supported-frontmatter type or not
- if not, force changing filename only
- if yes, allow the user to either
- still just change the filename, or
- additionally sync all frontmater properties,
be they already present or not.
>> - sync all information contained in the filename (id (date/time), title,>> keywords) to the frontmatter; if according properties do not exist,>> they are added at (e.g.) at the front, if they exist, they are kept>> at existing location but their values are replaced.>> (note: any other file's existing property remains unchanged)>> If more than one keyword with same key exists in the file>> (e.g. multiple #+keyword lines, then all but one are deleted).>> Note that the identifier should never be modified in the file name and,> by extension, should remain intact if shown in the front matter.> (Remember that Denote's linking only needs the file name, not the front> matter.) Changing the identifier is a recipe for breakage: links would> stop working and we could easily end up with lots of duplicates.>> I guess the idea of changing the identifier is to reflect a new date? I> think there are other ways to do that, such as by relying on file> attributes or, a bit more advanced setups, a Git (VCS in general)> history. For example, 'ls' accepts a '--sort' flag which can apply to> the time: <https://protesilaos.com/emacs/denote#h:a7fd5e0a-78f7-434e-aa2e-e150479c16e2>>> That aside, I agree with what you write there. We want to add what is> missing, replace old values, and delete duplicate entries.>
Yes, I agree: the 'identifier' should remain as unchangeable as
possible (by this I mean, that we can't prevent the user by
e.g. accidently changing it via some other filesystem operation, outside
of Denote).
Prohibiting the identifiers modification automatically means,
that any identifier syncing operation between filename and frontmatter
or the other way round will always copy the filename's identifier to the
frontmatter, never the other way round.
>> A 'reverse-rename' function would:>>>> - replace (rename) those parts of the filename, which have an existing>> correspondence in the frontmatter, i.e. title, keywords, date/time, identifier>> (if e.g. title does not exist as frontmatter, the filename's title>> remains unchanged, if it exists, the filename's title is updated>> accordingly).>> I like this idea! Though it should not touch the identifier, as> explained above.
Fully agreed!
>> Of course, we cannot prevent the user from editing whatever they want.> One can, for instance, rename the file with the standard renaming> facilities or wdired and edit the identifier. Any resulting errors> would not be our fault though. It is important for our purposes not to> cause breakage, as that jeopardises the trust users put in the package.
Same thought, as I noted above, yes!
>>> - there remains the problem, however, that #+idnetifier and #+date might>> not be in-sync themselves; a clear rule of preference would perhaps be a solution?>> Why is it a problem? The identifier is derived from the date because it> feels more natural to have, say, 20220727T060925 than apparent gibberish> like v7fd5e0a-78f7-434e-aa2e-a150479c16e9. The identifier is for> identification, but it remains sufficiently readable for practical> purposes.>> The #+date can be whatever the user wants. Perhaps we could even> provide or document a convenience function which appends a> #+lastmodified (or however we may call it) upon save? I am interested> to learn more about this and I am not against the idea.>> In my workflow, if I need to find files by date of modification, I> update Dired's 'ls' flags (check the link to the manual). If I need to> review the history of a file, I rely on Git. For example, the command> 'vc-print-log' (bound to 'C-x v l' by default), produces the log of> commits pertinent to the file. Typing 'd' in that buffer generates a> diff for the commit at point.>
I am unsure, whether I understand the purpose of the #+date: property
correctly:
I was under the impression, that it was just another, more human
readable way to display the date/time of the note creation - in other
words, an other representation of the 'identifier'.
(Because it also gets automatically created by denote on note-creation time)
If I understand you correctly, this is not the intention?
And the user may give its own meaning to this entry.
This is totally fine. In that case, it perhaps would make sense to state
in the documentation that the two properties initially represent the
same date (that of the note's creation), but that the #+date may be
freely changed thereafter and interpreted accordintly by the user?
[...]
>>> - Then the question arises: what is the 'sematics' of the #+date>> property in the frontmatter:>> a) is it just another (more human readable) way of the same thing>> as #+identifier, or>> b) is it meant to be some arbitrary date (having some arbitrary>> meaning to the author of the note, i.e. not necessarily meaning>> creation date of the note)?>> Depending on the answer (and semantic definition),>> #+date would be synced or not..>> As with all front matter, the #+date is for the user's convenience. It> reflects the time the note was created. It is not used for> identification purposes and performs no critical function in Denote's> code. Perhaps the user expects the #+date to show the last modification> time or something else? I am happy to support that use-case, provided> we understand what the needs are.
See my comment above on that matter.
[...]
So, to recap - now that I understand better Denote's inner workings:
----
'My ideal' rename-function would allow to either:
a) simply rename any file of ony type according to Denote's naming scheme
(this is already possible with the current implementation)
b) do the same with text files, even if their format (org, md,...) be
supported by Denote.
(unsure, whether this is currently supported or not)
c) do update and add - if missing - the frontmatter of such supported text files
('add' is not currently supported by 'denote-dired-rename-file')
For cases b) and c) frontmatter updating would occur only for
- identifier
- title
- keywords
(but not for date, if it already existed in the file's frontmatter)
Frontmatter adding would occur for all properties,
including date, in which case, the initial value set corresponds to that
of the identifier / the files creation.
'denote-dired-rename-file-and-add-front-matter' would then be redundant, no?
----
'My ideal' reverse-rename-function would modify the filename's
a) title, if #+title is present in the front matter
b) keywords, if #+filetags is present in the front matter
(edge case: multiple identical property names in frontmatter,
e.g. multiple #+filetags: some clear resolution behavior would be
required; same goes for the non-reverse 'rename' function above.)
----
>> All the best,> Prot
Same to you, Prot!
>> -- > Protesilaos Stavrou> https://protesilaos.com
> From: Hanspeter Gisler <hpgisleropen@bluewin.ch>> Date: Wed, 27 Jul 2022 16:35:38 +0200
Hello Hanspeter and thanks for taking the time to write this! I think
we are getting closer to figuring out how things stand. We shall then
move to the coding phase.
>>> b) the two are in-sync.>>>> Point b.>>>>> I personally did / would expect / want the two to be in-sync;>>> because at one time I want to find a file based e.g. on its keywords by means>>> of dired, another time I want to find it via grep.>>>> Having commented on the aforementioned renaming commands, I must ask:>> have they not worked at all for you?>> In my case, was confronted with the following situation:>> I had a bunch of .org files which I had created before starting to use> Denote and which I wanted to now be handled by Denote. >> They all had the following frontmatter:>> #+title: ...> #+date: ...> #+filetags: ...>> But of course no:> #+identifier:>> as this is Dentoe specific, I believe.
Yes, this is specific to Denote, as far as I can tell. For Org files,
we experimented with a PROPERTIES drawer that included an ID for a while
but ultimately scrapped those plans as they did not work for our
purposes. You can read the details in the change log of version 0.2.0.
The gist is: 'org-id' needs a database or equivalent.
<https://protesilaos.com/codelog/2022-07-04-denote-0-2-0/>
> So the 'denote-dired-rename-file' commnd did update correctly the first> 3 properties, but did not add the #+identifier: property.>> I am aware, that this is an edge case and this problem would not have> existed if I had created the original notes with Denote in the first> place. So once edited manually, the problem goes away.>> So, perhaps it is not justified to put a lot of effort into handling> this specific use-case...?
This may be an edge case, though it does not matter. It is a good
example of a scenario where complementing the front matter is the right
thing to do. More below.
> [...]>>>>>> What do you think of the following:>>>>>> A 'rename' function would either:>>>>>> - just change the filename>>> (either on users request or anyways, if the file's filetype>>> is unsupported by denotes frontmatter handling), or>>>> I am not sure I follow the parenthetic statement here. That granted, I>> will try to comment: the 'denote-dired-rename-file' renames any file and>> only rewrites the front matter (subject to a "yes-or-no" prompt) if it>> finds one.>> By this I meant:> If the rename function has the following functionalities> (similar to the currently existing 'denote-dired-rename-file'):>> - detect, whether the to be renamed file is of> denote-supported-frontmatter type or not > > - if not, force changing filename only>> - if yes, allow the user to either> - still just change the filename, or> - additionally sync all frontmater properties,> be they already present or not.
I see. Thanks for the explanation!
>>> - sync all information contained in the filename (id (date/time), title,>>> keywords) to the frontmatter; if according properties do not exist,>>> they are added at (e.g.) at the front, if they exist, they are kept>>> at existing location but their values are replaced.>>> (note: any other file's existing property remains unchanged)>>> If more than one keyword with same key exists in the file>>> (e.g. multiple #+keyword lines, then all but one are deleted).>>>> Note that the identifier should never be modified in the file name and,>> by extension, should remain intact if shown in the front matter.>> (Remember that Denote's linking only needs the file name, not the front>> matter.) Changing the identifier is a recipe for breakage: links would>> stop working and we could easily end up with lots of duplicates.>>>> I guess the idea of changing the identifier is to reflect a new date? I>> think there are other ways to do that, such as by relying on file>> attributes or, a bit more advanced setups, a Git (VCS in general)>> history. For example, 'ls' accepts a '--sort' flag which can apply to>> the time: <https://protesilaos.com/emacs/denote#h:a7fd5e0a-78f7-434e-aa2e-e150479c16e2>>>>> That aside, I agree with what you write there. We want to add what is>> missing, replace old values, and delete duplicate entries.>>>> Yes, I agree: the 'identifier' should remain as unchangeable as> possible (by this I mean, that we can't prevent the user by> e.g. accidently changing it via some other filesystem operation, outside> of Denote).>> Prohibiting the identifiers modification automatically means,> that any identifier syncing operation between filename and frontmatter> or the other way round will always copy the filename's identifier to the> frontmatter, never the other way round.
Very well! The file name identifier is the source of truth in this
case.
> [... 22 lines elided]>>>>> - there remains the problem, however, that #+idnetifier and #+date might>>> not be in-sync themselves; a clear rule of preference would perhaps be a solution?>>>> Why is it a problem? The identifier is derived from the date because it>> feels more natural to have, say, 20220727T060925 than apparent gibberish>> like v7fd5e0a-78f7-434e-aa2e-a150479c16e9. The identifier is for>> identification, but it remains sufficiently readable for practical>> purposes.>>>> The #+date can be whatever the user wants. Perhaps we could even>> provide or document a convenience function which appends a>> #+lastmodified (or however we may call it) upon save? I am interested>> to learn more about this and I am not against the idea.>>>> In my workflow, if I need to find files by date of modification, I>> update Dired's 'ls' flags (check the link to the manual). If I need to>> review the history of a file, I rely on Git. For example, the command>> 'vc-print-log' (bound to 'C-x v l' by default), produces the log of>> commits pertinent to the file. Typing 'd' in that buffer generates a>> diff for the commit at point.>>>> I am unsure, whether I understand the purpose of the #+date: property> correctly:>> I was under the impression, that it was just another, more human> readable way to display the date/time of the note creation - in other> words, an other representation of the 'identifier'.> (Because it also gets automatically created by denote on note-creation time)>> If I understand you correctly, this is not the intention?> And the user may give its own meaning to this entry.>> This is totally fine. In that case, it perhaps would make sense to state> in the documentation that the two properties initially represent the> same date (that of the note's creation), but that the #+date may be> freely changed thereafter and interpreted accordintly by the user?
Yes, the #+date is open to interpretation. The user can do whatever
they want with it.
We can explain it better in the manual. The #+date reflects the time of
creation, which coincides with the input that is used to derive the
#+identifier. If the user does not tamper with the front matter, the
two entries show the same underlying datum.
If, however, the user wants to change the date, they are free to do so.
It is not the #+identifier and Denote does not do anything with the
#+date.
The manual can explain all the technicalities, though I feel this is a
case of "less is more". We can explain what Denote needs to work and
then discuss how the front matter is for the user's convenience and/or
for external functionality (again, we take it seriously, want to keep it
up-to-date, and all that).
> [... 21 lines elided]> So, to recap - now that I understand better Denote's inner workings:>> ---->> 'My ideal' rename-function would allow to either:>> a) simply rename any file of ony type according to Denote's naming scheme> (this is already possible with the current implementation)>> b) do the same with text files, even if their format (org, md,...) be> supported by Denote.> (unsure, whether this is currently supported or not)> > c) do update and add - if missing - the frontmatter of such supported text files> ('add' is not currently supported by 'denote-dired-rename-file')>> For cases b) and c) frontmatter updating would occur only for> - identifier> - title> - keywords>> (but not for date, if it already existed in the file's frontmatter)
Point a is covered. Point b is also covered insofar as the file name is
concerned. Our renaming functions do not discriminate between regular
file types.
Point c is where we need more tweaks. We do not have a "complement"
operation, to fill in the missing entries in the front matter. You are
right that "add" is not included in 'denote-dired-rename-file', though
it is in 'denote-dired-rename-file-and-add-front-matter'.
1, The technical question is whether the "complement" and "add"
operations should be part of the same piece of functionality: (i)
complement the front matter with the missing entries, or (ii) if all
are missing add them. The whole action is subject to a confirmation
prompt. Does this mode of behaviour make sense?
2. We also have a question of ergonomics. Two commands are overkill.
They can confuse the user and are harder to explain. Can we
consolidate all three of the "rewrite", "complement", "add"
operations? If yes, then this is the right thing to do. Too many
options end up being to our detriment.
As for the #+date, I think it would be trivial to handle it as well once
we figure out the aforementioned. We are almost there.
> Frontmatter adding would occur for all properties,> including date, in which case, the initial value set corresponds to that> of the identifier / the files creation.>> 'denote-dired-rename-file-and-add-front-matter' would then be redundant, no?
Yes, that command should be deprecated. We can discuss the handling of
the #+date at a later point. I feel it is fairly easy in the grand
scheme of this change.
> ---->> 'My ideal' reverse-rename-function would modify the filename's>> a) title, if #+title is present in the front matter>> b) keywords, if #+filetags is present in the front matter
Yes, this makes perfect sense. Though before we add new functionality
we must refactor the existing one and test it thoroughly.
> (edge case: multiple identical property names in frontmatter,> e.g. multiple #+filetags: some clear resolution behavior would be> required; same goes for the non-reverse 'rename' function above.)
We can come up with clever heuristics, though I feel the simplest
solution will be the best for the project: just use the first #+filetags
we find. Duplicates of this sort are the user's fault (again, we will
test our renaming functions to make sure we don't produce such unwanted
output).
Please let me know what you think. We should clarify any remaining
points and then start working on the code.
All the best,
Prot
--
Protesilaos Stavrou
https://protesilaos.com
Hello!
I would like to add comments on the `reverse-rename` functionality
and other refactors.
About the syncing of the front matter with the filename, I think we
can only have it one way: from the front matter to the filename. This
is because of the sluggification process. There are constraints on the
characters allowed in the filename (and keywords), but they are in
their raw form in the front matter.
As a user, I would also like to be able to get my filename and front
matter in sync. For example, if I change the title or keywords in the
front matter, then run some `denote-update-filename` command, Denote
could take care of changing the filename appropriately, given the
front matter.
In fact, we could simplify all the rename functions to these two:
- `denote-add-front-matter` which adds a front-matter to a file
without changing the filename, and
- `denote-update-filename` which takes care of getting a valid
Denote filename, from the front matter (and current filename).
Then we can provide other convenience commands that perform the
previous 2 steps in one command and for batch operations.
In summary, this would be the path to get a Denote filename:
user input --> front matter --> valid file name
This is the `reverse-rename` function proposed above, but it could be
the standard way of Denote.
Concerning the #+identifier in the front matter, since it is not used
by Denote and it is not an interesting data to look at (the #+date is
more interesting), maybe we could get rid of it in the front matter
and only keep it in the filename? I think it would also reduce the
confusion about its similarity with the date. The doc would only have
to mention that the ID part of the filename must never be modified by
hand.
Does that make sense?
> From: Jean-Philippe Gagné Guay <jeanphilippe150@gmail.com>> Date: Wed, 27 Jul 2022 22:31:55 -0400>> Hello!>> I would like to add comments on the `reverse-rename` functionality> and other refactors.
Hello Jean-Philippe and thanks for writing this!
> About the syncing of the front matter with the filename, I think we> can only have it one way: from the front matter to the filename. This> is because of the sluggification process. There are constraints on the> characters allowed in the filename (and keywords), but they are in> their raw form in the front matter.
You are right that the sluggification leaves out characters. We need a
set of ordinary words as input which we use to convert into the
hyphenated title field in the file name. For the reverse rename this is
already given in the front matter.
You know this already, just writing it here for everyone's sake: our
renaming now prompts for a title, uses it as input for the
sluggification that goes into the file name, and then takes the user's
input and inserts it in the front matter.
My point being that we can have it both directions, given the requisite
arrangements.
> As a user, I would also like to be able to get my filename and front> matter in sync. For example, if I change the title or keywords in the> front matter, then run some `denote-update-filename` command, Denote> could take care of changing the filename appropriately, given the> front matter.
Yes, I think this reverse renaming is a good idea and we must make it
happen.
> In fact, we could simplify all the rename functions to these two:>> - `denote-add-front-matter` which adds a front-matter to a file> without changing the filename, and> - `denote-update-filename` which takes care of getting a valid> Denote filename, from the front matter (and current filename).
What would be the use-case of the former without the latter? Do you
mean it in the sense I explained in my last message to this thread?
Namely, where it would have a smart mechanism of (i) add the whole block
if all is absent, (ii) complement existing entries with the missing
ones, (iii) rewrite existing entries.
> Then we can provide other convenience commands that perform the> previous 2 steps in one command and for batch operations.
Agreed. Once we have the basic tools in place, we will work on the
ergonomics. The key is to simplify what we offer so that we don't have
to explain many different workflows. It should "just work".
> In summary, this would be the path to get a Denote filename:>> user input --> front matter --> valid file name>> This is the `reverse-rename` function proposed above, but it could be> the standard way of Denote.
I am fine with this. We would, of course, retain the functionality that
renames any file (e.g. PDF) without the front matter part.
> Concerning the #+identifier in the front matter, since it is not used> by Denote and it is not an interesting data to look at (the #+date is> more interesting), maybe we could get rid of it in the front matter> and only keep it in the filename? I think it would also reduce the> confusion about its similarity with the date. The doc would only have> to mention that the ID part of the filename must never be modified by> hand.
I think Hanspeter had a use for the #+identifier, such as for grep
operations. This was also my idea for its original inclusion. Maybe
there are other workflows as well that could use it? Perhaps external
tools?
Though yes, strictly speaking Denote does not need this datum to perform
its work. It is for the user's convenience.
> Does that make sense?
It does and thanks for joining the discussion!
All the best,
Prot
--
Protesilaos Stavrou
https://protesilaos.com
>> In fact, we could simplify all the rename functions to these two:>>>> - `denote-add-front-matter` which adds a front-matter to a file>> without changing the filename, and>> - `denote-update-filename` which takes care of getting a valid>> Denote filename, from the front matter (and current filename).> What would be the use-case of the former without the latter? Do you> mean it in the sense I explained in my last message to this thread?> Namely, where it would have a smart mechanism of (i) add the whole block> if all is absent, (ii) complement existing entries with the missing> ones, (iii) rewrite existing entries.
I don't have a use-case for the former without the latter, just for
the latter without the former! (The `denote-update-filename` would be
useful on its own.) I was thinking more about providing simple and
general tools to handle different use-cases. But maybe they should be
internal functions instead of user commands.
I am thinking that adding reverse-rename functions will lead to more
complexity, but I actually prefer them to the non-reverse functions. I
am trying to see Denote in another angle hoping to get something simpler
than just adding additional commands.
I am going to try to list the different use-cases and how a user could
handle them. We can expand this list to get a clearer picture.
(1) Modify title/keywords through the front matter and get the
filename updated.
Proposition: Run `denote-update-filename`.
(2) Get a file "my_note.org" to a valid Denote note.
Prop.: Run `denote-convert-to-denote` which would add a front
matter first, then derive the filename.
(3) Rename an existing (and valid) Denote note.
Prop.: Modify the front matter and run `denote-update-filename`.
This is different from `denote-dired-rename-file`, but it
is in a sense simpler because we don't have to handle all
the prompts of `denote-dired-rename-file` and we don't have
to confirm the changes with the user because he is already
modifying the file. One issue with this is that the user
will not see all the possible keywords to choose from, but
it might not be a big issue.
(4) Maybe given a valid Denote filename but no front matter,
add a front matter, but I am not sure...
Prop.: Run `denote-convert-to-denote` which would handle this case
as well.
(5) Have a Denote filename for a PDF.
Prop.: Run `denote-convert-to-denote`.
(6) Batch operations...
The more general idea would be to see the front matter as THE source of
truth from which to derive the filename.
> I think Hanspeter had a use for the #+identifier, such as for grep> operations. This was also my idea for its original inclusion. Maybe> there are other workflows as well that could use it? Perhaps external> tools?
Do you mean that the line `#+identifier: some_id` can be grep'ed to
retrieve a note? Maybe this workflow can be replaced by using `find` and
looking for a file starting with "some_id" instead.
Jean-Philippe
Protesilaos Stavrou <info@protesilaos.com> writes:
>> From: Hanspeter Gisler <hpgisleropen@bluewin.ch>>> Date: Wed, 27 Jul 2022 16:35:38 +0200>> Hello Hanspeter and thanks for taking the time to write this! I think> we are getting closer to figuring out how things stand. We shall then> move to the coding phase.>
Hello Prot, You are very welcome.
[...]
>> #+identifier:>>>> as this is Dentoe specific, I believe.>> Yes, this is specific to Denote, as far as I can tell. For Org files,> we experimented with a PROPERTIES drawer that included an ID for a while> but ultimately scrapped those plans as they did not work for our> purposes. You can read the details in the change log of version 0.2.0.> The gist is: 'org-id' needs a database or equivalent.>> <https://protesilaos.com/codelog/2022-07-04-denote-0-2-0/>
Thank you for the link, it is very illuminative.
[...]
>>> So the 'denote-dired-rename-file' commnd did update correctly the first>> 3 properties, but did not add the #+identifier: property.>>>> I am aware, that this is an edge case and this problem would not have>> existed if I had created the original notes with Denote in the first>> place. So once edited manually, the problem goes away.>>>> So, perhaps it is not justified to put a lot of effort into handling>> this specific use-case...?>> This may be an edge case, though it does not matter. It is a good> example of a scenario where complementing the front matter is the right> thing to do. More below.
Ok, great.
[...]
>> 'My ideal' rename-function would allow to either:>>>> a) simply rename any file of ony type according to Denote's naming scheme>> (this is already possible with the current implementation)>>>> b) do the same with text files, even if their format (org, md,...) be>> supported by Denote.>> (unsure, whether this is currently supported or not)>> >> c) do update and add - if missing - the frontmatter of such supported text files>> ('add' is not currently supported by 'denote-dired-rename-file')>>>> For cases b) and c) frontmatter updating would occur only for>> - identifier>> - title>> - keywords>>>> (but not for date, if it already existed in the file's frontmatter)>> Point a is covered. Point b is also covered insofar as the file name is> concerned. Our renaming functions do not discriminate between regular> file types.>> Point c is where we need more tweaks. We do not have a "complement"> operation, to fill in the missing entries in the front matter. You are> right that "add" is not included in 'denote-dired-rename-file', though> it is in 'denote-dired-rename-file-and-add-front-matter'.>> 1, The technical question is whether the "complement" and "add"> operations should be part of the same piece of functionality: (i)> complement the front matter with the missing entries, or (ii) if all> are missing add them. The whole action is subject to a confirmation> prompt. Does this mode of behaviour make sense?>
I think it would be very important, that a user
would not have to discriminate between "complement" and "add".
That is the responsability of the function; because:
- without first opening the file in a buffer the user does not now
whether there are no, some or all entries present as frontmatter.
(and the user should be enabled to do this w/o first having to
open the file.
- after all: the frontmatter of a file either having none, some or all
entries present is just a matter of 'frontmatter completeness'.
Any of these cases should be manageable indiscriminately by the user.
> 2. We also have a question of ergonomics. Two commands are overkill.> They can confuse the user and are harder to explain. Can we> consolidate all three of the "rewrite", "complement", "add"> operations? If yes, then this is the right thing to do. Too many> options end up being to our detriment.
I fully agree!
The user should not be forced to think about which function she has to
use to add / complete the frontmatter. This is the responsibility of the
function. After all, independent of the initial state of the
frontmatter's completeness (non, some or all entries are present),
the expected result is always: all entries are present in the frontmatter.
(if - of course - the user wants to add the frontmatter along at all)
The thing I am not sure of, is:
a) 1 rename function only? : which allows the user to chosse somehow between:
a1) rename filename without touching the file's content (frontmatter)
a2) rename filename with completing the file's frontmatter
or
b) 2 functions (a bit similar to the current situation (2 function),
where one implements only a1) and the other implements only a2).
>> As for the #+date, I think it would be trivial to handle it as well once> we figure out the aforementioned. We are almost there.>>> Frontmatter adding would occur for all properties,>> including date, in which case, the initial value set corresponds to that>> of the identifier / the files creation.>>>> 'denote-dired-rename-file-and-add-front-matter' would then be redundant, no?>> Yes, that command should be deprecated. We can discuss the handling of> the #+date at a later point. I feel it is fairly easy in the grand> scheme of this change.
ok.
>>> ---->>>> 'My ideal' reverse-rename-function would modify the filename's>>>> a) title, if #+title is present in the front matter>>>> b) keywords, if #+filetags is present in the front matter>> Yes, this makes perfect sense. Though before we add new functionality> we must refactor the existing one and test it thoroughly.
Agree!
>>> (edge case: multiple identical property names in frontmatter,>> e.g. multiple #+filetags: some clear resolution behavior would be>> required; same goes for the non-reverse 'rename' function above.)>> We can come up with clever heuristics, though I feel the simplest> solution will be the best for the project: just use the first #+filetags> we find. Duplicates of this sort are the user's fault (again, we will> test our renaming functions to make sure we don't produce such unwanted> output).
Yes, perfectly fine.
Most important then is, that the implemented behavior is well defined
and described in the documentation, so the user knows what to provide
and what to expect.
>> Please let me know what you think. We should clarify any remaining> points and then start working on the code.>> All the best,> Prot
Same to you, Prot.
>> -- > Protesilaos Stavrou> https://protesilaos.com