Hi,

A key to make your notes productive is to have several methods to 
find content. I just completed an early version of the 
Denote-Explore package. This package provides some functions to 
explore your denote collection:

1. Statistics
2. Take a random walk (snowball sampling)
3. QA functions (improve note quality)

Still to do: Network visualisation (graphical and perhaps in plain 
text)

https://github.com/pprevos/denote-explore

P.S. Prot, the random walk functions use some Denote private 
functions.

Love to read what you think.

regards

P:)

-- 
Dr Peter Prevos
---------------
peterprevos.com

> From: Peter Prevos <peter@prevos.net>
> Date: Fri, 20 Jan 2023 22:27:03 +1100
>
> Hi,

Hello Peter,

> A key to make your notes productive is to have several methods to 
> find content. I just completed an early version of the 
> Denote-Explore package. This package provides some functions to 
> explore your denote collection:
>
> 1. Statistics
> 2. Take a random walk (snowball sampling)
> 3. QA functions (improve note quality)

Sounds promising!

> Still to do: Network visualisation (graphical and perhaps in plain 
> text)

I think this is a feature that is in high demand.

> https://github.com/pprevos/denote-explore
>
> P.S. Prot, the random walk functions use some Denote private 
> functions.

Don't worry about using private functions.  What I did before was to set
up aliases so the new public function would still work with the old
private name, while warning the user of the change.

I will take a look at your code as soon as I can.  In principle, if
there is a need to use a private function from denote.el, it means that
we should make it public.

> Love to read what you think.

I will tell you more once I check the implementation details.

All the best,
Prot

-- 
Protesilaos Stavrou
https://protesilaos.com

Thanks Prot,

I am tinkering with visualising partial networks in plain text 
using the hierarchy package (show all links two-deep from active 
buffer). Step one is crating some Elisp functions to store the 
Denote network in a has table.

For more interactive graphical implementation one needs to go 
outside R. There are two options:

1. Build middleware so that Denote users can use the wonderful 
org-roam-ui package (perhaps after Emacs 29 lands with database 
support).
2. Build something in R using the igraph and shiny packages 
(simple ways to create a web interface)

Regards

P:)

Hi Peter
Nice work!

> 1. Build middleware so that Denote users can use the wonderful
> org-roam-ui package (perhaps after Emacs 29 lands with database
> support).

Denote works with Org-roam without a middleware for org files. I don not
mean to discourage you from developing a middleware with additional
features you wish to add. I would just like to point out that you have a
starting point that you can already play with.

All you need is to change the front matter configuration to use "id"
instead of Denote's default "identifier". Org-id will take the id
property. Denote's backlinks work without using "denote:" link type --
backlinks are searched with regexp for the id.

Optionally, you can set 'org-id-ts-format' to generate the same id as
Denote does (Org version 9.5 onward), so that you have options to use
either org(-roam) or denote as the starting point to writing a new note.

With my md-roam, markdown files can also be put into Org-roam's
database. Org-roam-ui works with markdown as well as org files. It is
what I use together with Denote and with no "denote:" link type.

Cheers, nobiot

Peter Prevos <peter@prevos.net> writes:

> Ideally, I like some Emacs code that converts the Denote folder to a
> JSON, like this:
>
> { "nodes": [
>      { "id": "20230123T085900", "name": "Denote example" },
>      { "id": "20230123T085930", "name": "A linked note" }
>    ],
>    "links": [
>      { "source": "20230123T085900", "target": "20230123T085930"}
>    ],
>    "tags": [
>     { "id": "20230123T085900", "tag": "denote" },
>     { "id": "20230123T085930", "tag": "denote" },
> }
>
> Looks like a bunch of association lists :)

Indeed. Since you are already making a hashtable, perhaps it would be
        just as easy as putting it to this function? (I believe your Emacs need
to be built with JSON support). Or build an alist and then put it into
the function.

    (json-serialize OBJECT &rest ARGS)

    Return the JSON representation of OBJECT as a string.

    OBJECT must be t, a number, string, vector, hashtable, alist, plist,
    or the Lisp equivalents to the JSON null and false values, and its
    elements must recursively consist of the same kinds of values.  t
    will be converted to the JSON true value.  Vectors will be converted
    to JSON arrays, whereas hashtables, alists and plists are converted
    to JSON objects.  Hashtable keys must be strings without embedded
    null characters and must be unique within each object.  Alist and
    plist keys must be symbols; if a key is duplicate, the first
    instance is used.

Hi Peter & nobiot

I'm interested in generating a graphical representation of my notes as
well.

Nobiot, you mentioned using Org-roam on Denote notes:

> All you need is to change the front matter configuration to use "id"
> instead of Denote's default "identifier". Org-id will take the id
> property. Denote's backlinks work without using "denote:" link type --
> backlinks are searched with regexp for the id.

Can you elaborate? The Org-roam database doesn't seem to pick up Denote
identifiers, even when they are indicated with `#+id:` in the
front-matter. Or do you mean to change the link type?

Cheers
EFLS

Elias Storms <elias.storms@gmail.com> writes:

> Nobiot, you mentioned using Org-roam on Denote notes:
>
>> All you need is to change the front matter configuration to use "id"
>> instead of Denote's default "identifier". Org-id will take the id
>> property. Denote's backlinks work without using "denote:" link type --
>> backlinks are searched with regexp for the id.
>
> Can you elaborate? The Org-roam database doesn't seem to pick up Denote
> identifiers, even when they are indicated with `#+id:` in the
> front-matter. Or do you mean to change the link type?

EFLS, happy to.  Instead of the keyword `#+id:`, you would need to use a
property drawer in this form at the top of the file before all the meta
data with `#+` -- that's how Org-ID works -- like this:

    :PROPERTIES:
    :ID:       20230130T173758
    :END:
    #+title:      This is an org created from denote
    #+date:       [2023-01-30 lun. 17:37]
    #+filetags:   :philosophy:
    #+identifier: 20230130T173758

If you are creating a new file via `denote`, you'd need to save the
buffer and run `org-roam-db-sync`. I have just tried it with `emacs -Q`
and it worked on my end.

For the denote front matter, something like this should work:

    (setq denote-org-front-matter
            ":properties:
        :ID: %4$s
        :end:
        #+title:      %1$s
        #+date:       %2$s
        #+filetags:   %3$s
        #+identifer:  %4$s
        \n")

You can keep the identifier like this example above. I'd remove it as it
is redundant and `denote-rename-file` and friends do not need it (also
tested now on my end).

There should be a more elegant way to automate adding new files to the
db, but I do not know a good way at the moment.  On my end, I use my own
`md-roam` to interface markdown files with `org-roam` and "it just
works" -- sorry, this is just a way to say "I just got off work and do
not have a mental capacity to remember / work out how I do this
exactly".

Happy to help you further if needed.

nobiot

Noboru Ota <me@nobiot.com> writes:
> If you are creating a new file via `denote`, you'd need to save the
> buffer and run `org-roam-db-sync`. I have just tried it with `emacs -Q`
> and it worked on my end.
> [...omission...]
> There should be a more elegant way to automate adding new files to the
> db, but I do not know a good way at the moment.  On my end, I use my own
> `md-roam` to interface markdown files with `org-roam` and "it just
> works" -- sorry, this is just a way to say "I just got off work and do
> not have a mental capacity to remember / work out how I do this
> exactly".

Sorry, a quick correction:

There is no need to run 'org-roam-db-sync' manually. Just need to set
'org-roam-directory' and 'denote-directory' to the same one (I didn't
note this explicitly earlier; it might be obvious, but just in
case).

I had not loaded 'org-roam' (and 'org-roam-db') when I created a new
file via denote. I have just re-tested it now; a newly created org file
gets put into org-roam-db automatically when you save the buffer if
'org-roam-db-autosync-mode' is active.  The presence of 'md-roam' is not
relevant.  You still need the id to be in the file's property drawer.

'denote' calls 'find-file' (internally inside 'denote--prepare-note') so
when 'org-roam-db-autosync-mode is active', this hook gets run on 'find-file'

    ;; org-roam-db
    (add-hook 'find-file-hook  #'org-roam-db-autosync--setup-file-h)

This then triggers a chain of setup for org files within
'org-roam-directory' = 'denote-directory' (via 'denote' or other means).

nobiot

Many thanks for the explanation! I have used Org-roam in the past so
your explanation makes sense. Adding the `ID` to the `:PROPERTIES:`
drawer is trivial.

Org-roam does pick up the Denote IDs this way, but unfortunately the default
`[[denote:id][Title]]` type of links aren't registered, resulting in a
graph without links. I probably need to use `id:` type links for this to
work.

So perhaps some middleware is needed? One that would copy the Denote
directory and modify each file so it has the property drawer and
replaces links. Not sure what's best here.

Cheers
EFLS

> From: Elias Storms <elias.storms@gmail.com>
> Date: Tue, 31 Jan 2023 14:28:03 +0100
>
> Many thanks for the explanation! I have used Org-roam in the past so
> your explanation makes sense. Adding the `ID` to the `:PROPERTIES:`
> drawer is trivial.
>
> Org-roam does pick up the Denote IDs this way, but unfortunately the default
> `[[denote:id][Title]]` type of links aren't registered, resulting in a
> graph without links. I probably need to use `id:` type links for this to
> work.
>
> So perhaps some middleware is needed? One that would copy the Denote
> directory and modify each file so it has the property drawer and
> replaces links. Not sure what's best here.

Notwithstanding those workarounds, I wonder if there is scope to just
tweak the upstream package to support 'file:' links for general Org
files?  Put differently, I don't see why the visualisation of
connections needs to have an opinion of how those are established.  The
'id:' is an internal detail to create resilient links, but they still
are "links".

I have no knowledge of the specifics and don't have time to do any
coding on this front, but this abstraction seems reasonable to me.  It
would then allow us to go the extra step of registering the 'denote:'
link type.  Perhaps the best way for the upstream project to do this is
to provide a user option whose default value is a regexp for 'file:' or
'id:' links.  The user can then add whatever link type they want in
addition to those.

-- 
Protesilaos Stavrou
https://protesilaos.com

Hi Prot

> I wonder if there is scope to just tweak the upstream package to
> support 'file:' links for general Org files?

Indeed, we'd need to look at the internals of Org-roam (but most of it
goes over my head).

I just took a look at the Org-roam manual, which states:

> 5.2 Links between Nodes

> We link between nodes using Org’s standard ID link (e.g. id:foo).
> While only ID links will be considered during the computation of links
> between nodes, Org-roam caches all other links in the documents for
> external use.

So it seems the links are already part of the Org-roam database, we'd
need to figure out how to make sure they are shown in the Org-roam UI.

It seems Peter already mentioned this approach on the Org-roam Github:
https://github.com/org-roam/org-roam-ui/issues/272

Anyway, it's something to explore further. Just wanted to mention these
things here for anyone looking at the same things.

All the best
EFLS

Elias Storms <elias.storms@gmail.com> writes:

> Org-roam does pick up the Denote IDs this way, but unfortunately the default
> `[[denote:id][Title]]` type of links aren't registered, resulting in a
> graph without links. I probably need to use `id:` type links for this to
> work.
>
> So perhaps some middleware is needed? One that would copy the Denote
> directory and modify each file so it has the property drawer and
> replaces links. Not sure what's best here.

No you do not need a middleware.

My simple suggestion is to *not* use 'denote:id' links, and simply use
the Org-ID's 'id:' links for Org files within 'org-roam-directory' =
'denote-directory'.  Denote's backlink functions just work without
'denote:id' because they use a regexp to find IDs (including this format
'[[id:<identifier>]]').

What is your reason for sticking to 'denote:id'?  I made the same
suggestion in a different context [1] discussing markdown and denote.
Similarly, if your notes stay within 'denote-directory' and use only Org
files, I do not find a compelling reason to use 'denote:id' at the
moment -- it may be useful if you want to have a link to a denote note
from the outide; I do not have this requirement.

[1]: https://lists.sr.ht/~protesilaos/denote/<86r0xozg89.fsf@nobiot.com>

Hi Nobiot

Thanks for the additional info. You're right that it is probably the
easiest to just use `id:` links, but I'd like to stick as close to the
default Denote config as possible for the time being. And moreover, as
you point out, the link type is required if you link to notes from
outside the denote-directory (which I often do).

The good news is that Org-roam UI can be modified to include the
`denote:` type links as Peter pointed out here:
https://github.com/org-roam/org-roam-ui/issues/272
It's as easy as redefining a function.

In any case, I'll experiment for a bit and see where it leads. Thanks
for pointing out how easy it is to leverage the Org-roam UI.

Have a nice day
EFLS

Elias Storms <elias.storms@gmail.com> writes:

> Indeed, we'd need to look at the internals of Org-roam (but most of it
> goes over my head).

If I were to make Org-roam work with the 'denote:' type link, I'd
override function 'org-roam-db-insert-link', so that the db could
register 'denote:' as an 'id:' Semantically both are IDs, so this is
logically consistent to me.

I suspect that changing this initial part might just do it (informed by
my experience of implementing Md-roam):

    (defun org-roam-db-insert-link (link)
      "Insert link data for LINK at current point into the Org-roam cache."
     (save-excursion
        (goto-char (org-element-property :begin link))
        (let ((type (org-element-property :type link))
              (path (org-element-property :path link))
              (source (org-roam-id-at-point))

For 'type', I'd consider adding something like this:

    (when type (string= type "denote") (setq type "id"))

'path' can stay as is, because it is an ID.

'source' can also stay, provided Org-ID with a property drawer is used
as we discussed.


—nobiot

Elias Storms <elias.storms@gmail.com> writes:

> The good news is that Org-roam UI can be modified to include the
> `denote:` type links as Peter pointed out here:
> https://github.com/org-roam/org-roam-ui/issues/272
> It's as easy as redefining a function.

Ah, our messages passed each other.  I see Peter has an easy workaround.
Sure; that's a good one, too.

You too have a good day.

—nobiot

> From: Elias Storms <elias.storms@gmail.com>
> Date: Tue, 31 Jan 2023 20:37:45 +0100

> [... 9 lines elided]

> The good news is that Org-roam UI can be modified to include the
> `denote:` type links as Peter pointed out here:
> https://github.com/org-roam/org-roam-ui/issues/272
> It's as easy as redefining a function.

This is what I had in mind.  As Peter writes, a user option that accepts
a lists of strings or regexp.  This would represent link types.  So a
user can add any link type they want.

-- 
Protesilaos Stavrou
https://protesilaos.com

Hi fellas,

The discussion on getting Denote stored in the Org Roam database 
is a bit strange to me. Why not just use Org Roam?

I am close to getting some pretty solid and simple code to 
visualise a Denote network (using R).

A working version is available here: 
https://github.com/pprevos/denote-explore

Still a bit of work to do...

Regards

P:)


-- 
Dr Peter Prevos
---------------
peterprevos.com

> The discussion on getting Denote stored in the Org Roam database
> is a bit strange to me. Why not just use Org Roam?

Peter, I share this sentiment, too. I guess, though, probably from a
different angle from yours. My suggestion is also "Why not just use Org
Roam?" for database and existing visualizing capability -- but of
course, another avenue you have been exploring is a great addition: "Why
not direcly use files created by Denote?".

Denote can be configured to create and work within normal Org files with
using Org-ID (instead of Denote-specific "denote:" link type in
Org). Org-roam can then store these normal Org files with the ID.

All of this is possible because they all have a good, composable
design. How to make use of this characteristic is up to you. Wonderful.

– nobiot