~protesilaos/denote

13 4

Connecting denote with bibliographies

Details
Message ID
<6add8bc63cab0a557fa4b9919e025afc@prevos.net>
DKIM signature
pass
Download raw message
Hi fellow denoters,

I have been experimenting with Denote a bit and like many others out 
there It could be a great alternative for Org roam.

One of the great features of Org roam is to connect notes with a 
bibliograpohy (BibTeX, Biblatex or CSL-JSON).

Several packages are available that make connections between notes and 
the bibliography: helm-bibtex, ivy-bibtex, ebib and citar. Citar seems 
ideally suited for denote as it leverages the same minibuffer completion 
functionality.

These packages achieve four main tasks:
- Provide access to your collections of bibliographies (searching, 
narrowing etc)
- Insert citations in various major modes
- Link to PDF documents for reading
- Link to a note file related to the entry

That last part is what I am looking for with Denote.

All the mentioned packages have a mechanism to recognise whether a 
bibliography entry has a note. Two approaches are used:
1. Recogniseable filename (in most cases it uses the citation key as 
filename)
2. Property inside the file

Packages like Citar can define helper functions to relates bibliography 
entries to notes. Ideally it would be filename-based, for example: 
"20220722T083612--protesilaos-2022-denote__bib_emacs.org" This would 
mean that this is a bibliographical fine note (bib tag) with 
protesilaos-2022-denote as citation tag.

I am using Denote as a stepping stone to learn Emacs lisp. Can anyone 
help me on the path to link Citar to Denote? Documentation: 
https://github.com/emacs-citar/citar#notes

perhaps somebody already has done this?

regards

P:)


-- 
Dr Peter Prevos
  ---------------
peterprevos.com
Details
Message ID
<87o7xh51bs.fsf@protesilaos.com>
In-Reply-To
<6add8bc63cab0a557fa4b9919e025afc@prevos.net> (view parent)
DKIM signature
pass
Download raw message
> From: Peter Prevos <peter@prevos.net>
> Date: Fri, 22 Jul 2022 08:39:46 +1000
>
> Hi fellow denoters,

Hello Peter,

> [... 26 lines elided]
>
> Packages like Citar can define helper functions to relates bibliography 
> entries to notes. Ideally it would be filename-based, for example: 
> "20220722T083612--protesilaos-2022-denote__bib_emacs.org" This would 
> mean that this is a bibliographical fine note (bib tag) with 
> protesilaos-2022-denote as citation tag.
>
> I am using Denote as a stepping stone to learn Emacs lisp. Can anyone 
> help me on the path to link Citar to Denote? Documentation: 
> https://github.com/emacs-citar/citar#notes
>
> perhaps somebody already has done this?

Citar looks like the right tool for this job.  I am now taking a look at
the code.  It seems there are some changes to the symbols that the
README references.

I think I will have a concrete answer for you later today.  Will keep
you (and this list) posted.

All the best,
Prot

-- 
Protesilaos Stavrou
https://protesilaos.com
Details
Message ID
<878rol4sce.fsf@protesilaos.com>
In-Reply-To
<87o7xh51bs.fsf@protesilaos.com> (view parent)
DKIM signature
pass
Download raw message
> From: Protesilaos Stavrou <info@protesilaos.com>
> Date: Fri, 22 Jul 2022 05:27:19 +0300
>
>> I am using Denote as a stepping stone to learn Emacs lisp. Can anyone 
>> help me on the path to link Citar to Denote? Documentation: 
>> https://github.com/emacs-citar/citar#notes
>>
>> perhaps somebody already has done this?
>
> Citar looks like the right tool for this job.  I am now taking a look at
> the code.  It seems there are some changes to the symbols that the
> README references.

I'm back!  There are a couple of typos in the README.  Those should be
"notes", not singular:

    and the =citar-register-note-source= and =citar-remove-note-source=
    convenience functions.

Anyhow, they are not relevant for now.

> I think I will have a concrete answer for you later today.  Will keep
> you (and this list) posted.

I took a look at the code.  It seems that the default value of
'citar-notes-sources' does the right job of gathering file paths,
testing that a key is present in them, and the like.

What I'm missing is documentation on what this facility is supposed to
do.  I need insight into the actual workflow because now I don't get how
the notes and the bibliography would work in tandem.  Citar's README
assumes prior knowledge which I lack.

Maybe you have a .bib file that includes something like the following?

    @article{KEY-TO-DENOTE-FILE-HERE,
      title={Testing Denote},
      author={Protesilaos},
      journal={The Journal of Incessant Testing},
      volume={27},
      number={2},
      pages={97--111},
      year={2022},
      publisher={Test Publisher Press}
    }

And this is a reference to a note in the 'denote-directory'?  The latter
is part of 'citar-notes-paths'?  Do the Bib entries require a specific
key or, generally, do they have a certain anatomy we need to conform
with?

Note that I am not familiar with LaTeX, BibTeX, and friends.  I can find
my way around the Elisp code, but otherwise need help with the non-Elisp
parts.

-- 
Protesilaos Stavrou
https://protesilaos.com
Saša Janiška <sjaniska@atmarama.net>
Details
Message ID
<87h739f2i2.fsf@atmarama.ddns.net>
In-Reply-To
<87o7xh51bs.fsf@protesilaos.com> (view parent)
DKIM signature
missing
Download raw message
Hello,

just to chime in saying I'm also *very* interested to be able to 
connect Denote 
with bibligraphies - probably by keeping bibliographic database in 
e.g. Zotero 
and then exporting it via "Better Bibtex for Zotero" 
(https://github.com/retorquere/zotero-better-bibtex).

Hopefully, more academic users will discover Denote and help come 
to the right 
workflow...

Sincerely,
Saša

-- 
A person who is not disturbed by the incessant flow of
desires — that enter like rivers into the ocean, which is
ever being filled but is always still — can alone achieve
peace, and not the man who strives to satisfy such desires.
Details
Message ID
<87bkthhv3j.fsf@protesilaos.com>
In-Reply-To
<87h739f2i2.fsf@atmarama.ddns.net> (view parent)
DKIM signature
pass
Download raw message
> From: Saša Janiška <sjaniska@atmarama.net>
> Date: Fri, 22 Jul 2022 20:01:25 +0200
>
> Hello,

Hi Saša,

> just to chime in saying I'm also *very* interested to be able to
> connect Denote with bibligraphies - probably by keeping bibliographic
> database in e.g. Zotero and then exporting it via "Better Bibtex for
> Zotero" (https://github.com/retorquere/zotero-better-bibtex).

In principle, I am eager to do what is necessary and/or help however I
can.  Though as I noted earlier in this thread, I am not familiar with
any of those tools.  Their docs all assume prior knowledge, which I
lack.  Thus my ability to be of immediate service is severely limited.

> Hopefully, more academic users will discover Denote and help come to
> the right workflow...

Sure.  Gaining insight into the workflow will help a lot, as then it is
easier to understand what kind of glue code we need.

All the best,
Prot

-- 
Protesilaos Stavrou
https://protesilaos.com
Fernando de Morais <fernandodemorais.jf@gmail.com>
Details
Message ID
<87y1v1dtdi.fsf@sekai.mail-host-address-is-not-set>
In-Reply-To
<87bkthhv3j.fsf@protesilaos.com> (view parent)
DKIM signature
pass
Download raw message
Hello everyone!

Thanks for this great package. I've been using Denote for a very short
time, but I'm enjoying and being more productive with it.

-----

Protesilaos Stavrou <info@protesilaos.com> writes:

>> just to chime in saying I'm also *very* interested to be able to
>> connect Denote with bibligraphies - probably by keeping bibliographic
>> database in e.g. Zotero and then exporting it via "Better Bibtex for
>> Zotero" (https://github.com/retorquere/zotero-better-bibtex).
>
> In principle, I am eager to do what is necessary and/or help however I
> can.  Though as I noted earlier in this thread, I am not familiar with
> any of those tools.  Their docs all assume prior knowledge, which I
> lack.  Thus my ability to be of immediate service is severely limited.


Regarding the discussion, for Ebib[1], at least, there is a more clear
path to integrate Denote to it. If we use the ``Notes Files'' section of
the manual[2] as a reference, it can be seen that Ebib:

  - Can store one note per file: `ebib-notes-storage';

  - Applies a custom name to the notes, based on the values of
    `ebib-notes-name-transform-function';

  - Creates new notes based on a template configured with
    `ebib-notes-template';

  - Searches for the value of a unique identifier with a function,
    `ebib-notes-extract-org-ids', that try to get the `:CUSTOM_ID:'
    property;

  - It can use `org-capture' to create notes (although it needs a
    specific function in the template): `ebib-notes-use-org-capture'.

The section as a whole is very informative on the subject and contains a
greater level of detail on these points that I mentioned briefly.

One downside, maybe, is that Ebib notes system only works with Org
files.  I believe that by using other formats its capabilities to show
notes for the given entry will become non-functional, unless hacked.


Protesilaos Stavrou <info@protesilaos.com> writes:

>> Hopefully, more academic users will discover Denote and help come to
>> the right workflow...
>
> Sure.  Gaining insight into the workflow will help a lot, as then it is
> easier to understand what kind of glue code we need.


Well, at least for me, creating notes with Denote from Ebib is a great
way to compose metanotes for a specific entry.  Allow me to elaborate
briefly:

  I organize my library with Ebib and access my books through it.  Then,
  I use Denote through `org-capture' to write notes based on my
  readings.

  That way, when the time to create a metanote arises, I just need to
  access Ebib, go to the desired entry and hit the key that will bring
  `ebib-popup-note' well integrated with Denote.  😀

  This metanote will be visible in the `external note' field and can be
  accessed quickly through Ebib's interface.

I hope something in this email can be helpful to move this
discussion/feature implementation forward.


Footnotes:

[1]  https://melpa.org/#/ebib
[2]  https://joostkremers.github.io/ebib/ebib-manual.html#notes-files

-- 
Regards,
Fernando de Morais.


P.S.: Saša, In my workflow, I was able to completely migrate from Zotero
     to Ebib without losing anything in terms of productivity or
     organization.  As it works directly with BibTeX files, an
     intermediate plugin is not necessary.  If I can make a suggestion,
     give it a try.  Conducting bibliographic research using only Emacs
     allowed me to gain productivity.  👍
Details
Message ID
<87ilm5f6bz.fsf@protesilaos.com>
In-Reply-To
<87y1v1dtdi.fsf@sekai.mail-host-address-is-not-set> (view parent)
DKIM signature
pass
Download raw message
> From: Fernando de Morais <fernandodemorais.jf@gmail.com>
> Date: Sat, 03 Sep 2022 00:33:29 -0300
>
> Hello everyone!

Hello Fernando!

> Thanks for this great package. I've been using Denote for a very short
> time, but I'm enjoying and being more productive with it.

You are welcome!

> -----
>
> Protesilaos Stavrou <info@protesilaos.com> writes:
>
>>> just to chime in saying I'm also *very* interested to be able to
>>> connect Denote with bibligraphies - probably by keeping bibliographic
>>> database in e.g. Zotero and then exporting it via "Better Bibtex for
>>> Zotero" (https://github.com/retorquere/zotero-better-bibtex).
>>
>> In principle, I am eager to do what is necessary and/or help however I
>> can.  Though as I noted earlier in this thread, I am not familiar with
>> any of those tools.  Their docs all assume prior knowledge, which I
>> lack.  Thus my ability to be of immediate service is severely limited.
>
>
> Regarding the discussion, for Ebib[1], at least, there is a more clear
> path to integrate Denote to it. If we use the ``Notes Files'' section of
> the manual[2] as a reference, it can be seen that Ebib:
>
>   - Can store one note per file: `ebib-notes-storage';
>
>   - Applies a custom name to the notes, based on the values of
>     `ebib-notes-name-transform-function';
>
>   - Creates new notes based on a template configured with
>     `ebib-notes-template';
>
>   - Searches for the value of a unique identifier with a function,
>     `ebib-notes-extract-org-ids', that try to get the `:CUSTOM_ID:'
>     property;
>
>   - It can use `org-capture' to create notes (although it needs a
>     specific function in the template): `ebib-notes-use-org-capture'.
>
> The section as a whole is very informative on the subject and contains a
> greater level of detail on these points that I mentioned briefly.
>
> One downside, maybe, is that Ebib notes system only works with Org
> files.  I believe that by using other formats its capabilities to show
> notes for the given entry will become non-functional, unless hacked.

This is promising.  I read the section and am eager to start
experimenting with it despite the lacunae in my knowledge regarding
LaTeX and friends.  Do you have a minimal reproducible setup that I
could use?  This will help me understand the workflow and test the code.

The fact that Ebib wants Org files is not a problem per se.  It is a
third-party package and we are not making any compromises on Denote's
end.  The code can 'let' bind the file type to use Org, or make such
adjustments.  We will figure it out.

> Protesilaos Stavrou <info@protesilaos.com> writes:
>
>>> Hopefully, more academic users will discover Denote and help come to
>>> the right workflow...
>>
>> Sure.  Gaining insight into the workflow will help a lot, as then it is
>> easier to understand what kind of glue code we need.
>
>
> Well, at least for me, creating notes with Denote from Ebib is a great
> way to compose metanotes for a specific entry.  Allow me to elaborate
> briefly:
>
>   I organize my library with Ebib and access my books through it.  Then,
>   I use Denote through `org-capture' to write notes based on my
>   readings.
>
>   That way, when the time to create a metanote arises, I just need to
>   access Ebib, go to the desired entry and hit the key that will bring
>   `ebib-popup-note' well integrated with Denote.  😀
>
>   This metanote will be visible in the `external note' field and can be
>   accessed quickly through Ebib's interface.
>
> I hope something in this email can be helpful to move this
> discussion/feature implementation forward.

Yes, this is useful and it inspires me to take it to the next step.
Thank you!  Once I have something in place, I can share code on this
list and elicit feedback from you and others on how best to proceed.

All the best,
Protesilaos (or simply "Prot")

-- 
Protesilaos Stavrou
https://protesilaos.com
Fernando de Morais <fernandodemorais.jf@gmail.com>
Details
Message ID
<87ilm5dnta.fsf@sekai.mail-host-address-is-not-set>
In-Reply-To
<87ilm5f6bz.fsf@protesilaos.com> (view parent)
DKIM signature
pass
Download raw message
Protesilaos Stavrou <info@protesilaos.com> writes:

> This is promising.  I read the section and am eager to start
> experimenting with it despite the lacunae in my knowledge regarding
> LaTeX and friends.  Do you have a minimal reproducible setup that I
> could use?  This will help me understand the workflow and test the code.

Sure! Follows the content of BibTeX file with three entries (save as
.bib).

<--------------------------------->

@Book{balasubramanian14:_human_hand_inspir_robot_hand_devel,
	copyright = {Copyright \textcopyright{} 2014 Springer International Publishing Switzerland. All rights reserved.},
	series = {Springer Tracts in Advanced Robotics},
	isbn = {978-3-319-03017-3},
	language = {english},
	doi = {10.1007/978-3-319-03017-3},
	abstract = {"The Human Hand as an Inspiration for Robot Hand Development" presents an edited collection of authoritative contributions in the area of robot hands.},
	edition = {First},
	address = {Cham},
	volume = {95},
	year = {2014},
	publisher = {Springer International Publishing},
	title = {The Human Hand as an Inspiration for Robot Hand Development},
	editor = {Ravi Balasubramanian and Veronica J. Santos}
}

@Book{moore18:_clinic_orien_anatom,
	address = {Philadelphia},
	copyright = {Copyright \textcopyright{} 2018 Wolters Kluwer. All rights reserved.},
	isbn = {978-1-4963-4721-3},
	language = {english},
	abstract = {The world’s most trusted clinically focused anatomy textRenowned for comprehensive coverage, the best-selling Clinically Oriented Anatomy guides students from initial anatomy and foundational science courses through clinical training and practice. The eighth edition reflects significant New information and updates and maintains the highest standards for scientific and clinical accuracy.},
	edition = {Eighth},
	year = {2018},
	publisher = {Wolters Kluwer},
	title = {Clinically Oriented Anatomy},
	author = {Keith L. Moore and Arthur F. Dalley II and Anne M. R. Agur}
}

<--------------------------------->

It can be used in conjunction with Ebib for your tests. A most minimal
setup that I can describe might be the following:

  - After installing Ebib, please customize `ebib-default-directory' and
    place the sample BibTeX file there;

  - `M-x ebib';

  - At Ebib-index buffer (the one in the smaller window at the top),
    press `o' (`ebib-open-bibtex-file') and select the sample .bib file;

  - You can now operate on the database, edit entries with `e'
    (`ebib-edit-entry') and compose notes with `N' (`ebib-popup-note');

  - `z' (`ebib-leave-ebib-windows') delete the windows and `q'
    (`ebib-quit') quits the mode completely.

Please let me know if this is enough or if I can help with more details.

> The fact that Ebib wants Org files is not a problem per se.  It is a
> third-party package and we are not making any compromises on Denote's
> end.  The code can 'let' bind the file type to use Org, or make such
> adjustments.  We will figure it out.

Very good news!

> Yes, this is useful and it inspires me to take it to the next step.
> Thank you!  Once I have something in place, I can share code on this
> list and elicit feedback from you and others on how best to proceed.

I'm glad to know that my previous email has helped to bring new ideas to
bear on this matter. Happy to help! 🤝

-- 
Regards,
Fernando de Morais.
Details
Message ID
<87wnakn2bf.fsf@protesilaos.com>
In-Reply-To
<87ilm5dnta.fsf@sekai.mail-host-address-is-not-set> (view parent)
DKIM signature
pass
Download raw message
> From: Fernando de Morais <fernandodemorais.jf@gmail.com>
> Date: Sat, 03 Sep 2022 02:33:37 -0300

Hello again Fernando,

[ I am including Peter in Cc as this may be relevant for Citar+Denote. ]

> Protesilaos Stavrou <info@protesilaos.com> writes:
>
>> This is promising.  I read the section and am eager to start
>> experimenting with it despite the lacunae in my knowledge regarding
>> LaTeX and friends.  Do you have a minimal reproducible setup that I
>> could use?  This will help me understand the workflow and test the code.
>
> Sure! Follows the content of BibTeX file with three entries (save as
> .bib).
>
> [... 33 lines elided]
>
> It can be used in conjunction with Ebib for your tests. A most minimal
> setup that I can describe might be the following:
>
>   - After installing Ebib, please customize `ebib-default-directory' and
>     place the sample BibTeX file there;
>
>   - `M-x ebib';
>
>   - At Ebib-index buffer (the one in the smaller window at the top),
>     press `o' (`ebib-open-bibtex-file') and select the sample .bib file;
>
>   - You can now operate on the database, edit entries with `e'
>     (`ebib-edit-entry') and compose notes with `N' (`ebib-popup-note');
>
>   - `z' (`ebib-leave-ebib-windows') delete the windows and `q'
>     (`ebib-quit') quits the mode completely.
>
> Please let me know if this is enough or if I can help with more details.

I got Ebib to work thanks to your instructions and sample data.  Here is
a prototype to use Denote-style file paths in Ebib's note-creating
function:

    (defun my-ebib-denote-transform-key (key)
      "Return Denote-style path with KEY as the title.
    Used for `ebib-notes-name-transform-function'."
      (if (null ebib-notes-directory)
          (user-error "Please set the value of `ebib-notes-directory'")
        (file-name-sans-extension ; Denote has to add ".org" but ebib adds it too
         (file-name-nondirectory
          (denote--path
           key
           '("ebib") ; example with hardcoded keywords
           ebib-notes-directory
           (format-time-string denote--id-format)
           'org)))))

    (setq ebib-notes-name-transform-function #'my-ebib-denote-transform-key)

I am using it with those settings:

    (setq ebib-default-directory "/home/prot/test-files/testing-ebib/")
    (setq ebib-notes-directory "/home/prot/test-files/testing-ebib-notes/") ; make it differ from `ebib-file-search-dirs'

Some notes:

1. The KEY in Bib includes characters that Denote considers "illegal"
   for use in the file name.  For example, in the data you sent me, we
   find this: pons08:_wearab_robot.  The colon is not allowed anywhere,
   while the underscore is reserved for Denote's keywords (the manual
   explains the rationale).

2. Denote thus removes those "illegal" characters and constructs a file
   path which uses Bib's KEY as the title component.  Like this:
   20220903T134420--pons08-wearab-robot__ebib.org

   pons08:_wearab_robot ==> pons08-wearab-robot

   Is the transformation acceptable?  It is just for the file name, not
   the contents of the newly created note.

3. My function currently hardcodes the keywords that Denote adds to the
   file path.  This is a list of strings which currently has only the
   single "ebib" string.  You can edit this accordingly.

4. I tried to use the familiar keywords' prompt of Denote instead of a
   hardcoded value, but this leads to recursion.  I even had to kill my
   Emacs process, which I don't recall ever doing before.  Maybe
   something in Ebib does not like ebib-notes-name-transform-function to
   call its own interactive functions.  I still need to investigate
   this.

5. My sample code does not add the familiar Denote front matter.
   Instead, it lets Ebib do its thing.  This is because (i) the
   expectation of ebib-notes-name-transform-function is that only a path
   to a file name is returned, (ii) Denote does not actually need the
   front matter for its own links.  Having the front matter is good for
   the user, but not strictly necessary.  Just pointing this out as the
   Denote+Ebib integration may still benefit from adding (how?) our
   front matter to those notes.

6. Note that I am using 'denote--path' with some wrapper functions to
   accommodate Ebib's expectations.  The double hyphens indicate that
   the code is for internal use and may change without further notice.
   However, as the next version of Denote will be 1.0.0, this specific
   function and a few others can be made "public" (i.e. 'denote-path').
   Doing so tells the user/developer that we fully support it, are
   committed to not break it, and/or will warn about any changes to it.

If you try this, please let me know (i) if it works for you, (ii) if
there are areas that can be improved, (iii) any other considerations
pertaining to the workflow.

Bear in mind that Denote can happily work with multiple standalone
directories of notes.  We call them "silos".  It is explained in the
manual, but I can also elaborate here.  The point is that the
'ebib-notes-directory' does not need to be the same as one's main
'denote-directory'.

-- 
Protesilaos Stavrou
https://protesilaos.com
Details
Message ID
<87fsh8vdoj.fsf@prevos.net>
In-Reply-To
<87wnakn2bf.fsf@protesilaos.com> (view parent)
DKIM signature
pass
Download raw message
Hi Prot et al.,

On your first two points: Using the citation key in the filename 
is not an option with Denote due to the strict naming convention, 
as Prot pointed out. There is no such convention for BibTeX keys, 
which can take almost any form. I have thousands of references and 
changing the keys is not an option.

This means that the citation key has to be somewhere in the front 
matter for the bibliography software to link the note to the 
BibTeX entry. Emacs has:

- Ebib (special interface)
- Helm / Ivy BibTeX
- Citar (minibuffer completion)

For this purpose I am experimenting with citar: 
https://github.com/pprevos/denote-citar

The next four weeks I am dedicating to solving this problem and 
learn more Elisp along the way.

Regards

P:)

Protesilaos Stavrou <info@protesilaos.com> writes:

>> From: Fernando de Morais <fernandodemorais.jf@gmail.com>
>> Date: Sat, 03 Sep 2022 02:33:37 -0300
>
> Hello again Fernando,
>
> [ I am including Peter in Cc as this may be relevant for 
> Citar+Denote. ]
>
>> Protesilaos Stavrou <info@protesilaos.com> writes:
>>
>>> This is promising.  I read the section and am eager to start
>>> experimenting with it despite the lacunae in my knowledge 
>>> regarding
>>> LaTeX and friends.  Do you have a minimal reproducible setup 
>>> that I
>>> could use?  This will help me understand the workflow and test 
>>> the code.
>>
>> Sure! Follows the content of BibTeX file with three entries 
>> (save as
>> .bib).
>>
>> [... 33 lines elided]
>>
>> It can be used in conjunction with Ebib for your tests. A most 
>> minimal
>> setup that I can describe might be the following:
>>
>>   - After installing Ebib, please customize 
>>   `ebib-default-directory' and
>>     place the sample BibTeX file there;
>>
>>   - `M-x ebib';
>>
>>   - At Ebib-index buffer (the one in the smaller window at the 
>>   top),
>>     press `o' (`ebib-open-bibtex-file') and select the sample 
>>     .bib file;
>>
>>   - You can now operate on the database, edit entries with `e'
>>     (`ebib-edit-entry') and compose notes with `N' 
>>     (`ebib-popup-note');
>>
>>   - `z' (`ebib-leave-ebib-windows') delete the windows and `q'
>>     (`ebib-quit') quits the mode completely.
>>
>> Please let me know if this is enough or if I can help with more 
>> details.
>
> I got Ebib to work thanks to your instructions and sample data. 
> Here is
> a prototype to use Denote-style file paths in Ebib's 
> note-creating
> function:
>
>     (defun my-ebib-denote-transform-key (key)
>       "Return Denote-style path with KEY as the title.
>     Used for `ebib-notes-name-transform-function'."
>       (if (null ebib-notes-directory)
>           (user-error "Please set the value of 
>           `ebib-notes-directory'")
>         (file-name-sans-extension ; Denote has to add ".org" but 
>         ebib adds it too
>          (file-name-nondirectory
>           (denote--path
>            key
>            '("ebib") ; example with hardcoded keywords
>            ebib-notes-directory
>            (format-time-string denote--id-format)
>            'org)))))
>
>     (setq ebib-notes-name-transform-function 
>     #'my-ebib-denote-transform-key)
>
> I am using it with those settings:
>
>     (setq ebib-default-directory 
>     "/home/prot/test-files/testing-ebib/")
>     (setq ebib-notes-directory 
>     "/home/prot/test-files/testing-ebib-notes/") ; make it 
>     differ from `ebib-file-search-dirs'
>
> Some notes:
>
> 1. The KEY in Bib includes characters that Denote considers 
> "illegal"
>    for use in the file name.  For example, in the data you sent 
>    me, we
>    find this: pons08:_wearab_robot.  The colon is not allowed 
>    anywhere,
>    while the underscore is reserved for Denote's keywords (the 
>    manual
>    explains the rationale).
>
> 2. Denote thus removes those "illegal" characters and constructs 
> a file
>    path which uses Bib's KEY as the title component.  Like this:
>    20220903T134420--pons08-wearab-robot__ebib.org
>
>    pons08:_wearab_robot ==> pons08-wearab-robot
>
>    Is the transformation acceptable?  It is just for the file 
>    name, not
>    the contents of the newly created note.
>
> 3. My function currently hardcodes the keywords that Denote adds 
> to the
>    file path.  This is a list of strings which currently has 
>    only the
>    single "ebib" string.  You can edit this accordingly.
>
> 4. I tried to use the familiar keywords' prompt of Denote 
> instead of a
>    hardcoded value, but this leads to recursion.  I even had to 
>    kill my
>    Emacs process, which I don't recall ever doing before.  Maybe
>    something in Ebib does not like 
>    ebib-notes-name-transform-function to
>    call its own interactive functions.  I still need to 
>    investigate
>    this.
>
> 5. My sample code does not add the familiar Denote front matter.
>    Instead, it lets Ebib do its thing.  This is because (i) the
>    expectation of ebib-notes-name-transform-function is that 
>    only a path
>    to a file name is returned, (ii) Denote does not actually 
>    need the
>    front matter for its own links.  Having the front matter is 
>    good for
>    the user, but not strictly necessary.  Just pointing this out 
>    as the
>    Denote+Ebib integration may still benefit from adding (how?) 
>    our
>    front matter to those notes.
>
> 6. Note that I am using 'denote--path' with some wrapper 
> functions to
>    accommodate Ebib's expectations.  The double hyphens indicate 
>    that
>    the code is for internal use and may change without further 
>    notice.
>    However, as the next version of Denote will be 1.0.0, this 
>    specific
>    function and a few others can be made "public" 
>    (i.e. 'denote-path').
>    Doing so tells the user/developer that we fully support it, 
>    are
>    committed to not break it, and/or will warn about any changes 
>    to it.
>
> If you try this, please let me know (i) if it works for you, 
> (ii) if
> there are areas that can be improved, (iii) any other 
> considerations
> pertaining to the workflow.
>
> Bear in mind that Denote can happily work with multiple 
> standalone
> directories of notes.  We call them "silos".  It is explained in 
> the
> manual, but I can also elaborate here.  The point is that the
> 'ebib-notes-directory' does not need to be the same as one's 
> main
> 'denote-directory'.


-- 
Dr Peter Prevos
---------------
peterprevos.com
Fernando de Morais <fernandodemorais.jf@gmail.com>
Details
Message ID
<87v8q4p71z.fsf@sekai.mail-host-address-is-not-set>
In-Reply-To
<87wnakn2bf.fsf@protesilaos.com> (view parent)
DKIM signature
pass
Download raw message
Hello Protesilaos and Peter,

Protesilaos Stavrou <info@protesilaos.com> writes:

> If you try this, please let me know (i) if it works for you, (ii) if
> there are areas that can be improved, (iii) any other considerations
> pertaining to the workflow.

Thanks for the quick response. I tested your code and on the points you
asked to consider:

(i) The creation of the notes with Denote file name worked very well!
However, the integration with Ebib has not yet been completely satisfied
using only this function, because it was not able to find the notes;

(ii) Reading the manual and the code in `ebib-notes.el' library, I
realized that when `ebib-notes-storage' is set to `one-file-per-note',
only the filename is relevant to find it. Ebib uses the `K' specifier
function of the variable `ebib-notes-template-specifiers' to compose the
same `CUSTOM_ID:' generated when the note was created and looks for it
in the filename (as seen in the `ebib--notes-locate-note'). Perhaps
these are the starting points for improving the integration between the
packages;

(iii) For my specific case, I would like to have the front matter,
because it is very useful to me when I have to go back to old notes.

In the next few days I will be trying to contribute with some code
related to this specific integration, in order to cooperate with the
solution. Unfortunately, due to my workload, I won't be able to get
started right away, but I'll be back as soon as possible with some ideas
and some Elisp.

Thanks once more!

To Peter:

Citar is a very nice package indeed! I can clearly see a workflow that
complements Ebib with its great set of functionality.

Perhaps this discussion can show us a common way to integrate Denote
with both packages so that it is easy for one to access the notes
created by the other.

-- 
Regards,
Fernando de Morais.
Details
Message ID
<87tu5nn45z.fsf@protesilaos.com>
In-Reply-To
<87v8q4p71z.fsf@sekai.mail-host-address-is-not-set> (view parent)
DKIM signature
pass
Download raw message
> From: Fernando de Morais <fernandodemorais.jf@gmail.com>
> Date: Sat, 03 Sep 2022 16:54:48 -0300
>
> Hello Protesilaos and Peter,

Thank you Fernando for the feedback!

> Protesilaos Stavrou <info@protesilaos.com> writes:
>
>> If you try this, please let me know (i) if it works for you, (ii) if
>> there are areas that can be improved, (iii) any other considerations
>> pertaining to the workflow.
>
> Thanks for the quick response. I tested your code and on the points you
> asked to consider:
>
> (i) The creation of the notes with Denote file name worked very well!
> However, the integration with Ebib has not yet been completely satisfied
> using only this function, because it was not able to find the notes;
>
> (ii) Reading the manual and the code in `ebib-notes.el' library, I
> realized that when `ebib-notes-storage' is set to `one-file-per-note',
> only the filename is relevant to find it. Ebib uses the `K' specifier
> function of the variable `ebib-notes-template-specifiers' to compose the
> same `CUSTOM_ID:' generated when the note was created and looks for it
> in the filename (as seen in the `ebib--notes-locate-note'). Perhaps
> these are the starting points for improving the integration between the
> packages;

I see.  This is expected.  Ebib does not know how to find Denote notes
because it uses a different mechanism that we do (we search for the
identifier in the given directory).

> (iii) For my specific case, I would like to have the front matter,
> because it is very useful to me when I have to go back to old notes.

Okay.  It may then be easier to leverage Denote's existing ability to
use the contents of the active region as the default value of the title.
Try this to see what I mean:

- Mark a region which includes "pons08:_wearab_robot" (without the
  quotes).

- M-x denote

- Notice that the minibuffer prompt is

      "File title [pons08-wearab-robot]:"

- The bracketed text is the default value.  If you type RET without
  further input, it will be used as the note's title.  Then you get the
  prompt for keywords and you end up with a fully fledged Denote note,
  with front matter and all.

We can use this principle to have a command that is like ordinary
'denote' but which uses the region's text as the value of a special key.
That special key can be the Custom_id or whatever we want.  This way we
can create notes outside the Ebib mechanisms but which still behave as
Ebib notes.

I have not tested this, but think it will work.  It gives us maximum
control to optimise the function and to make no awkward compromises
(e.g. my previous code could not prompt for keywords).

> In the next few days I will be trying to contribute with some code
> related to this specific integration, in order to cooperate with the
> solution. Unfortunately, due to my workload, I won't be able to get
> started right away, but I'll be back as soon as possible with some ideas
> and some Elisp.

That's okay.  If we can implement the integration in an Ebib-agnostic
yet Ebib-compliant way, we have a general feature that Peter can/may
also use with denote-citar (mutatis mutandis).

> Thanks once more!

You are welcome!

> To Peter:
>
> Citar is a very nice package indeed! I can clearly see a workflow that
> complements Ebib with its great set of functionality.
>
> Perhaps this discussion can show us a common way to integrate Denote
> with both packages so that it is easy for one to access the notes
> created by the other.

I think the sentiment in this final paragraph is key.  If we have a
general note-taking command that can pass an arbitrary value to a
specified key/format, we have what we need to create file contents that
conform with the expectations of other packages.  The actual code is
easy, as we have all the elements in place and can add whatever trivial
details.

What is difficult, is to try to create Denote-style notes through the
functionality of other packages, because we then need to make
compromises (such as with my prototype from earlier).

[ If this is too abstract, I can provide actual code to let you see my
  idea in action. ]

-- 
Protesilaos Stavrou
https://protesilaos.com
Fernando de Morais <fernandodemorais.jf@gmail.com>
Details
Message ID
<87sfl7pixk.fsf@sekai.mail-host-address-is-not-set>
In-Reply-To
<87tu5nn45z.fsf@protesilaos.com> (view parent)
DKIM signature
pass
Download raw message
Protesilaos Stavrou <info@protesilaos.com> writes:

> Thank you Fernando for the feedback!

Hello Protesilaos. You are welcome!

> [ If this is too abstract, I can provide actual code to let you see my
>   idea in action. ]

If you can provide some code, I'll test as soon as possible and return
to you any pertinent considerations from my use case with Ebib.

Thanks!

-- 
Regards,
Fernando de Morais.
Details
Message ID
<87a67fml46.fsf@protesilaos.com>
In-Reply-To
<87sfl7pixk.fsf@sekai.mail-host-address-is-not-set> (view parent)
DKIM signature
pass
Download raw message
> From: Fernando de Morais <fernandodemorais.jf@gmail.com>
> Date: Sun, 04 Sep 2022 06:50:31 -0300

> [... 7 lines elided]

>> [ If this is too abstract, I can provide actual code to let you see my
>>   idea in action. ]
>
> If you can provide some code, I'll test as soon as possible and return
> to you any pertinent considerations from my use case with Ebib.
>
> Thanks!

Here is the PROTOTYPE:

    (defvar my-denote-org-front-matter-for-ebib
      ":PROPERTIES:
    :Custom_id:   %s
    :END:
    #+title:      %s
    #+date:       %s
    #+filetags:   %s
    #+identifier: %s
    \n"
      "Like `denote-org-front-matter' but with PROPERTIES drawer.")

    (defun my-denote-format-front-matter (bibkey title date keywords id)
      "Front matter for new notes.

    BIBKEY, TITLE, DATE, KEYWORDS, FILENAME, ID are all provided by
    `my-denote-ebib'."
      (let ((title (denote--format-front-matter-title title 'org))
            (kws (denote--format-front-matter-keywords keywords 'org)))
        (format my-denote-org-front-matter-for-ebib bibkey title date kws id)))

    (defun my-denote-prepare-ebib-note (bibkey title keywords date id directory)
      "Prepare a new note file.
    Arguments BIBKEY, TITLE, KEYWORDS, DATE, ID, and DIRECTORY are
    passed by `my-denote-ebib'."
      (let* ((path (denote--path title keywords directory id 'org))
             (buffer (find-file path))
             (header (my-denote-format-front-matter
                      bibkey
                      title
                      (format-time-string denote--id-format date)
                      keywords
                      id)))
        (with-current-buffer buffer
          (insert header))))

    (defun my-denote-ebib-prompt (&optional default-bib-key)
      "Read Bib key for `denote'.
    With optional DEFAULT-BIB-KEY use it as the default value."
      (let* ((def default-bib-key)
             (format (if (and def (not (string-empty-p def)))
                         (format "Bib key [%s]: " def)
                       "Bib key: ")))
        (read-string format nil nil def)))

    (defvar my-denote-directory-ebib "/home/prot/absolute/path/to/ebib/notes/")

    (defun my-denote-ebib (&optional beg end)
      (interactive)
      (let* ((defkey (when (use-region-p)
                       (buffer-substring-no-properties
                        (region-beginning)
                        (region-end))))
             (bibkey (my-denote-ebib-prompt defkey))
             (title (denote--title-prompt))
             (keywords (denote--keywords-prompt))
             (date (current-time))
             (id (format-time-string denote--id-format date))
             (directory my-denote-directory-ebib))
        (denote--barf-duplicate-id id)
        ;; (message "%s %s %s %s %s %s" bibkey title keywords date id directory)
        (my-denote-prepare-ebib-note bibkey title keywords date id directory)))

What this does:

- The command 'my-denote-ebib' is like the regular 'denote' in that it
  asks for a title and keywords.  But it first prompts for a BIBKEY.
  This specific prompt will use the text of the highlighted region, if
  available.

- The variable 'my-denote-org-front-matter-for-ebib' contains the format
  that is used for those notes.  Notice that it has a :Custom_id:.  The
  value of BIBKEY is added there verbatim.  You can, of course, change
  the Custom_id to whatever you want.  This is just to give you an idea.

- The variable 'my-denote-directory-ebib' is where your notes should be
  stored.  You can use 'denote-directory' as its value, if that is what
  you want.  I am keeping it separate for this demo.

- The other functions just prepare the front matter.

Using the command 'my-denote-ebib' (i.e. not the command of Ebib), I get
this file:

    20220904T142557--this-is-a-test-for-ebib__denote_ebib_testing.org

It has the following front matter:

    :PROPERTIES:
    :Custom_id:   some:key_here1-2-+3=4
    :END:
    #+title:      This is a test for Ebib
    #+date:       20220904T142557
    #+filetags:   :denote:ebib:testing:
    #+identifier: 20220904T142557


I just added an arbitrary value at the BIBKEY prompt to show you that
this is not transforming the value in any way.

Do you think this file and front matter combination moves things in the
right direction?  Can Ebib now find the note if it has the desired
PROPERTIES drawer?  What do you want to see improved in terms of
workflow?

[ Note that I am indenting the code here for readability.  But make sure
  to evaluate the 'my-denote-org-front-matter-for-ebib' variable without
  the indentation, so that it is not read as part of the string. ]

-- 
Protesilaos Stavrou
https://protesilaos.com
Reply to thread Export thread (mbox)