I implemented something similar to this arena for shared memory. A
file in /dev/shm is mmap()ed into a big address space slot and
allocation takes place within it (Linux has no problem mmapping a file
smaller than the address range - access beyond current size just
generates SIGBUS). Whenever bumping the allocation pointer would
exceed the current file size it is extended by another chunk. Using
atomics makes it all lock-free for concurrent access. A possible race
condition is avoided by using fallocate() rather than ftruncate() to
extend the file as the latter can only increase the file size. A
little python library understands the same format, which is very
useful for debugging and tweaking. Use a memfd if you don't want it to
outlive the creating process. It is still accessible externally
through /proc/PID/fd/NN
The data structure within this file is quite similar to the one you
describe in "An easy-to-implement, arena-friendly hash map". One
difference is that I use the xxHash function by the incomparable Yann
Collet. It is probably faster and better than anything you or I could
come up with and has an implementation using vector intrinsics.
Interesting, thanks for sharing, Oren! And a salient point about fallocate
vs. ftruncate. I haven't needed to consider that trade-off before. I also
like that trick of Python sharing data structure access. All new ideas for
me to consider.