~gpcf/advtrains-discuss

4 3

Thoughts on running Advanced Trains in async environment

Details
Message ID
<72b19de1-2415-4fb0-98e8-c8e6d4c02380@1f616emo.xyz>
DKIM signature
pass
Download raw message
I am thinking of running the main logic of advtrains in the async 
thread. Random thoughts:

Run all logic (i.e. path calculation, interlocking, train movement, 
train collision) in the async environment, and every time it returns to 
the main thread, sync the calculated results with the world (i.e. 
spawn/move train entities, sync passive component states), perform 
sync-only actions (e.g. detect/damage collided entities, halt trains 
colliding with walls), and sync data back into async (i.e. apply node DB 
changes).

This would be a challenging and compatibility-breaking change (e.g. some 
on_step calls such as texture changes are async-friendly, but some 
others such as vacuuming items up are not). But by doing this, we no 
longer add lag to the globalstep with heavy calculations.

As a shorter-term improvement, we may move saving data into the async 
environment (which I recently did on areas 
https://github.com/minetest-mods/areas/pull/83/commits/4064964135f1b152387f07675d4af55dbd7e6808).
Details
Message ID
<87msipy4nq.fsf@knoblauch.gpcf.eu>
In-Reply-To
<72b19de1-2415-4fb0-98e8-c8e6d4c02380@1f616emo.xyz> (view parent)
DKIM signature
pass
Download raw message
1F616EMO <root@1f616emo.xyz> writes:

> Run all logic (i.e. path calculation, interlocking, train movement,
> train collision) in the async environment, and every time it returns
> to the main thread, sync the calculated results with the world
> (i.e. spawn/move train entities, sync passive component states),
> perform sync-only actions (e.g. detect/damage collided entities, halt
> trains colliding with walls), and sync data back into async
> (i.e. apply node DB changes).

Honestly, the whole running the train simulation in a separate thread
kind of calls for running the stuff in a separate process written in a
different programming language, there was a bit of talk about an
advtrainsd written in C, however, with rust maturing over the years I'd
almost write the daemon in Rust. Then the advtrains mod would just be a
frontend, and could in theory be exchanged for other frontends such as a
2.5D Simutrans-like UI. The question is who writes that, and advtrains's
use case kind of needs bug-for-bug compatibility with the current lua
version of the mod.
Details
Message ID
<d5c66ce2-1916-4a9e-bc9a-cdf4dd71d05b@protonmail.com>
In-Reply-To
<87msipy4nq.fsf@knoblauch.gpcf.eu> (view parent)
DKIM signature
pass
Download raw message
> Run all logic (i.e. path calculation, interlocking, train movement, 
> train collision) in the async environment, and every time it returns to 
> the main thread, sync the calculated results with the world (i.e. 
> spawn/move train entities, sync passive component states), perform 
> sync-only actions (e.g. detect/damage collided entities, halt trains 
> colliding with walls), and sync data back into async (i.e. apply node DB 
> changes).

Minetest currently does not support multithreading, and the async env 
should not be used for multithreading according to this comment on Github:
https://github.com/minetest/minetest/pull/14659#issuecomment-2136019562

> Honestly, the whole running the train simulation in a separate thread
> kind of calls for running the stuff in a separate process written in a
> different programming language, there was a bit of talk about an
> advtrainsd written in C, however, with rust maturing over the years I'd
> almost write the daemon in Rust. Then the advtrains mod would just be a
> frontend, and could in theory be exchanged for other frontends such as a
> 2.5D Simutrans-like UI.

I'm currently somewhat skeptical about this, given how tightly various 
parts of Advtrains are put together. There are also things where a clean 
separation of a daemon and the MT frontend is not trivial, e.g. blocking 
trains with nodes.

> As a shorter-term improvement, we may move saving data into the async 
> environment (which I recently did on areas 
> https://github.com/minetest-mods/areas/pull/83/commits/4064964135f1b152387f07675d4af55dbd7e6808).

This would be doable, especially given that serialize_lib can be 
improved to, among other things, use core.safe_file_write instead of 
writing to the file directly (which also makes unittests easier; we can 
then use string comparison instead of writing to temporary files and 
reading from there). The same goes for serialization of the node database.

That said, I am not sure about the performance benefits here. We will 
likely still run the serialization code in the main environment, so this 
will only address the I/O bottleneck.

In terms of performance benefits, I added annotations for Tracy (which 
Minetest now supports in 5.10.0-dev) in the "tracy" branch. The main 
performance issue appears to be mainly related to (unsurprisingly) the 
LZB and path systems, but apparently there are certain helper functions 
(IIRC advtrains.get_rail_info_at and related) used by these systems that 
take up quite some time while not really doing much. I did not take much 
time to look deeply into the issue though, for various reasons.
Details
Message ID
<87iktdxsth.fsf@knoblauch.gpcf.eu>
In-Reply-To
<d5c66ce2-1916-4a9e-bc9a-cdf4dd71d05b@protonmail.com> (view parent)
DKIM signature
pass
Download raw message
"Y. Wang" <y5nw@protonmail.com> writes:
> This would be doable, especially given that serialize_lib can be 
> improved to, among other things, use core.safe_file_write instead of 
> writing to the file directly (which also makes unittests easier; we can 
> then use string comparison instead of writing to temporary files and 
> reading from there). The same goes for serialization of the node database.

btw, has anyone checked the performance of serialize_lib vs the minetest
json serializer?


Also, the whole advtrains performance issue is not as urgent as it used
to be, as with the LF server migration server load dropped dramatically
and advtrains runs pretty smoothly these days.
Details
Message ID
<DC91EEAD-C49C-4102-80D8-8A6F2DBD7706@1f616emo.xyz>
In-Reply-To
<87iktdxsth.fsf@knoblauch.gpcf.eu> (view parent)
DKIM signature
pass
Download raw message
Perhaps my concern was caused by my poor hardware. I plan to upgrade my server on Black Friday, and I’ll see how things goes after that.

> gpcf <gpcf@gpcf.eu>於2024年10月28日 03:15寫道:
> 
> 
> "Y. Wang" <y5nw@protonmail.com> writes:
>> This would be doable, especially given that serialize_lib can be
>> improved to, among other things, use core.safe_file_write instead of
>> writing to the file directly (which also makes unittests easier; we can
>> then use string comparison instead of writing to temporary files and
>> reading from there). The same goes for serialization of the node database.
> 
> btw, has anyone checked the performance of serialize_lib vs the minetest
> json serializer?
> 
> 
> Also, the whole advtrains performance issue is not as urgent as it used
> to be, as with the LF server migration server load dropped dramatically
> and advtrains runs pretty smoothly these days.
Reply to thread Export thread (mbox)