~protesilaos/denote

4 3

denote's scalability

Saša Janiška <sjaniska@atmarama.net>
Details
Message ID
<87sfmtf7im.fsf@atmarama.ddns.net>
DKIM signature
missing
Download raw message
Hello,

I'm very pleased with what I see/hear in regard to Denote...first 
start
watching SystemCrafter's presentation and then watched Prot's one.

File naming scheme, depending on other tools, file-format 
agnosticism
etc. are making Denote future-proof, but I have few questions- 
each one
in a separate thread...

I wonder if Denote is tested within an environment with some 
larger
number of tools, let's say > 10K?

In my search for an optimum tool. I was exploring some which are 
not
based on Emacs. like Zettlr, Joplin, Logseq...so, knowing well 
that
having one within Emacs' ecosystem is a great advantage, but still
wonder if it can scale for a serious note-taker/writer hoping to 
produca
large knowledge base?


Sincerely,
Saša

-- 
The intricacies of action are very hard to understand.
Therefore one should know properly what action is,
what forbidden action is, and what inaction is.
Details
Message ID
<878rolaxxu.fsf@protesilaos.com>
In-Reply-To
<87sfmtf7im.fsf@atmarama.ddns.net> (view parent)
DKIM signature
pass
Download raw message
> From: Saša Janiška <sjaniska@atmarama.net>
> Date: Fri, 22 Jul 2022 18:10:09 +0200
>
> Hello,

Hello Saša,

> I'm very pleased with what I see/hear in regard to Denote...first
> start watching SystemCrafter's presentation and then watched Prot's
> one.

And thanks to David (SystemCrafters) for doing it, by the way.

> File naming scheme, depending on other tools, file-format agnosticism
> etc. are making Denote future-proof, but I have few questions- each
> one in a separate thread...
>
> I wonder if Denote is tested within an environment with some larger
> number of tools, let's say > 10K?

I have not tested it with that high number.  I tried with 5k and
everything worked fine.  Though it depends on the specifics.  There
certainly comes a point where things will slow down.  Not just with
Denote, but Emacs in general, as well as with 'find', 'grep', and
friends.

Without willing to downplay the significance of this issue, I wonder
what kind of workflow will yield 10k notes and over what period of time.
Are those part of a usable, high quality knowledge base?  Or are they
the equivalent of a TODO for the agenda or maybe an Internet browser's
bookmark that you click once and forget about it?

My point is that I would personally reflect more on the workflow than on
whether the underlying tool is optimised for bigger data sets.  It is
easy to collect information.  The difficult part is to organise it and
take good care of it.

Again though, I am not downplaying the importance of this issue.

> In my search for an optimum tool. I was exploring some which are not
> based on Emacs. like Zettlr, Joplin, Logseq...so, knowing well that
> having one within Emacs' ecosystem is a great advantage, but still
> wonder if it can scale for a serious note-taker/writer hoping to
> produca large knowledge base?

I am not familiar with those and cannot comment on them.  For Emacs, we
already have org-roam which, I believe, can handle large data sets, by
virtue of its database-driven cache.  So it really depends on one's
needs.

About the "serious note-taker/writer" part, I am genuinely interested to
learn more about how that works.  Which are the particular demands.
What kind of daily output are we talking about?  And the like.

All the best,
Prot

-- 
Protesilaos Stavrou
https://protesilaos.com
Saša Janiška <sjaniska@atmarama.net>
Details
Message ID
<87o7xhf345.fsf@atmarama.ddns.net>
In-Reply-To
<878rolaxxu.fsf@protesilaos.com> (view parent)
DKIM signature
missing
Download raw message
Protesilaos Stavrou <info@protesilaos.com> writes:

Hello Prot,

> And thanks to David (SystemCrafters) for doing it, by the way.

:-)

> I have not tested it with that high number.  I tried with 5k and
> everything worked fine.

That's good to hear!

> Though it depends on the specifics.  There certainly comes a 
> point
> where things will slow down.  Not just with Denote, but Emacs in
> general, as well as with 'find', 'grep', and friends.

I see...


>
> Without willing to downplay the significance of this issue, I 
> wonder
> what kind of workflow will yield 10k notes and over what period 
> of time.

Well, if we're considering periord of 10 years, it means less than 
3
notes per day.

> Are those part of a usable, high quality knowledge base?

Yes.

> Or are they the equivalent of a TODO for the agenda or maybe an
> Internet browser's bookmark that you click once and forget about 
> it?

At the moment I do use `nb` (https://github.com/xwmx/nb/) as 
bookmark
manager, although I plan to somehow convert it to org-mode.

> My point is that I would personally reflect more on the workflow 
> than on
> whether the underlying tool is optimised for bigger data sets. 
> It is
> easy to collect information.  The difficult part is to organise 
> it and
> take good care of it.

Well, I'm studying Vedic philosophy and regularly lecturing on
it. Besides that I'm doing serious study of Vedic astrology 
(Jyotish)
and have to regularly consult many cllassic books on the
subject. Moreover, when working with people's readings I'd like to
collect interesting notes which do make ties between theory & 
practice.

So, personally I'm speaking about real world example and not just
theoretical idea. Moreover, while studying the subject there is 
also
plenty of opportunities to take many "fleeting notes" which are 
waiting
to be incorporated into personal knowledge base.

> Again though, I am not downplaying the importance of this issue.

Fair enough. ;)

> I am not familiar with those and cannot comment on them.  For 
> Emacs, we
> already have org-roam which, I believe, can handle large data 
> sets, by
> virtue of its database-driven cache.  So it really depends on 
> one's
> needs.

I've working setup for org-roam, but to be honest, intriscic 
simplicity
of your tool is tempting.  I alos higly appreaciate your care for 
end
users by providing high class documentation and building on the 
"top of
giants shoulders". :-)


> About the "serious note-taker/writer" part, I am genuinely 
> interested to
> learn more about how that works.  Which are the particular 
> demands.
> What kind of daily output are we talking about?  And the like.

I hope you could get some insight into my desired workflow, 
although I
can try to put together more thoughts.

As far as the output is concerned it is not my primary focus, 
although I
want to dive more into writing, since at the moment I'm mostly
expressing myself through speaking.

However, even for just lecturing, I want to take advantage of my
personal knowledge base to get deeper insight into the topics I'm
studying.

Last, but not the least, as some (spiritual) teacher put together: 
"In
the age in which we are living, to remember mean to writei it 
down!"


Sincerely,
Saša

p.s. Some other possibly related topics I'll touch by replying to 
either
existing thread(s) or creating new ones...

-- 
A person is said to be elevated in yoga when, having renounced
all material desires, he neither acts for sense gratification
nor engages in fruitive activities.
Details
Message ID
<87edydhve8.fsf@protesilaos.com>
In-Reply-To
<87o7xhf345.fsf@atmarama.ddns.net> (view parent)
DKIM signature
pass
Download raw message
> From: Saša Janiška <sjaniska@atmarama.net>
> Date: Fri, 22 Jul 2022 19:35:08 +0200
>
>> Though it depends on the specifics.  There certainly comes a point
>> where things will slow down.  Not just with Denote, but Emacs in
>> general, as well as with 'find', 'grep', and friends.
>
> I see...

My understanding is that as the scale increases, you start making
adjustments.  For example, you create new directories, merge files,
rewrite information to keep what is still relevant, use more
sophisticated search/regexp to narrow results, and the like.

Put differently, I am interested to know how one would approach this
issue with standard Unix tools.  I have never worked with such a large
sample size.

I am curious if anyone else reading this has another insight on the
matter.

If it can be done with Unix, then we can do it with Denote without
compromising on what we have.

> [... 18 lines elided]
>
>> My point is that I would personally reflect more on the workflow than
>> on whether the underlying tool is optimised for bigger data sets.  It
>> is easy to collect information.  The difficult part is to organise it
>> and take good care of it.
>
> Well, I'm studying Vedic philosophy and regularly lecturing on
> it. Besides that I'm doing serious study of Vedic astrology (Jyotish)
> and have to regularly consult many cllassic books on the
> subject. Moreover, when working with people's readings I'd like to
> collect interesting notes which do make ties between theory &
> practice.
>
> So, personally I'm speaking about real world example and not just
> theoretical idea. Moreover, while studying the subject there is also
> plenty of opportunities to take many "fleeting notes" which are
> waiting to be incorporated into personal knowledge base.

All clear.  Thanks for explaining it!

>> I am not familiar with those and cannot comment on them.  For Emacs,
>> we already have org-roam which, I believe, can handle large data
>> sets, by virtue of its database-driven cache.  So it really depends
>> on one's needs.
>
> I've working setup for org-roam, but to be honest, intriscic
> simplicity of your tool is tempting.  I alos higly appreaciate your
> care for end users by providing high class documentation and building
> on the "top of giants shoulders". :-)

Fair enough!  I just mention org-roam because its database seems better
suited for larger-scale operations (I presume).

>> About the "serious note-taker/writer" part, I am genuinely interested
>> to learn more about how that works.  Which are the particular
>> demands.  What kind of daily output are we talking about?  And the
>> like.
>
> I hope you could get some insight into my desired workflow, although I
> can try to put together more thoughts.

I did.  Thank you!

-- 
Protesilaos Stavrou
https://protesilaos.com
Details
Message ID
<9f7ad9190c405d967142f00248e0e356@prevos.net>
In-Reply-To
<87sfmtf7im.fsf@atmarama.ddns.net> (view parent)
DKIM signature
pass
Download raw message
Hi Saša,

I have written some code that generates 2000 random notes and I noticed 
no performance issues.

You can download them here: 
https://nest.greenant.net/index.php/s/d8f3xgGM8sSkMo7

I can share the code (in the R language) if you like to create more test 
notes.

Regards

P:)

---
Dr Peter Prevos
  ---------------
peterprevos.com

On 23-07-2022 02:10, Saša Janiška wrote:
> Hello,
> 
> I'm very pleased with what I see/hear in regard to Denote...first start
> watching SystemCrafter's presentation and then watched Prot's one.
> 
> File naming scheme, depending on other tools, file-format agnosticism
> etc. are making Denote future-proof, but I have few questions- each one
> in a separate thread...
> P:)P:)P:)P:)P:)P:)P:)P:)P:)P:)P:)
> I wonder if Denote is tested within an environment with some larger
> number of tools, let's say > 10K?
> 
> In my search for an optimum tool. I was exploring some which are not
> based on Emacs. like Zettlr, Joplin, Logseq...so, knowing well that
> having one within Emacs' ecosystem is a great advantage, but still
> wonder if it can scale for a serious note-taker/writer hoping to 
> produca
> large knowledge base?
> 
> 
> Sincerely,
> Saša
Reply to thread Export thread (mbox)