Wanted to bring this up at the Fennel user group meeting for some time,
but was unable to attend to it, so here's the spil in a text form.
For some time I've been working on a library for making HTTP requests[1]
asynchronously from Fennel without relying on anything but luasocket. I
have a library for asynchronous programming async.fnl[2] that implements
a channel-based workflow, and uses debug.sethook for scheduling. I've
used this library in fnl-http, and faced a certain problem - how do I
bundle async.fnl in such a way that fnl-http can still be a single file
library for ease of use while making sure that users can still use
async.fnl without being affected by the bundled version?
Here's the problem in a bit more detail.
In the async.fnl I have a check for determining if an object is a
Channel. It's a simple check for the object's contains an entry
identical to a Channel singleton, defined in the library:
(local Channel
{:dirty-puts 0 :dirty-takes 0})
(fn chan? [obj]
"Test if `obj` is a channel."
(match obj {:type Channel} true _ false))
Now, what happens, if a user loads a library? The dependency is just a
single file, so it can potentially be placed anywhere in the project.
For example:
(local async (require "async"))
(local some-channel (async.chan))
Now the user wants to use the fnl-http library in the same project.
Fnl-http is a bit different, as it is also a single-file, self-contained
library, but only if it is compiled to Lua, as there are quite a few
modules in it:
(local http (require "fnl-http"))
Now, the user tries to supply the channel as a body:
(http.post "some.url" {:body some-channel})
And gets an error, that the body type Channel is not supported, even
though the library says that it is in the readme. What's happening
here?
If we look at the package.loaded, we'll see two entries for async.fnl:
{"async" {...} ; the one that the user loaded explicitly
"lib.async" {...}} ; the one defined in package.preload of fnl-http
Because of that, there are two different Channel singletons and fnl-http
kinda expects the one from "lib.async". So the correct way of using
both libraries would be:
(local http (require "fnl-http"))
(local async (require "lib.async"))
(local some-channel (async.chan))
;; ----8<----
(http.post "some.url" {:body some-channel})
Now everything works as expected.
However, besides the singleton problem, there's actually another, more
dangerous problem. The async.fnl library uses debug.sethook, as I
mentioned, to implement a scheduler. This hook is only set when an
operation on the channel is pending. And if the user has both fnl-http
and async.fnl loaded, as in the first example, even if they never use
the channel-as-a-body feature of fnl-http, every time they use a
channel, or do an asynchronous HTTP request, they essentially have two
schedulers competing with eachother. The async.fnl library is designed
in such a way that this should not be a problem, but I can't guarantee
that, to be honest.
So, in order to fix this issue, I decided to provide a curated way of
installing the libraries, and ensuring that they're required properly.
Many of my repositories now feature the tasks/install script, that can
be invoked as:
fennel tasks/install --prefix /path/to/your/project/libs
In the case of fnl-http, the library will be installed to
/path/to/your/project/libs/io/gitlab/andreyorst/fnl-http.lua. The best
part of it is that inside this fnl-http.lua file, the async.fnl is
already preloaded as "io.gitlab.andreyorst.async". So if the user
installs the async.fnl via the install script, it would also end up in
/path/to/your/project/libs/io/gitlab/andreyorst/async.fnl.
What's left is to add this directory to the package and fennel paths
with:
fennel --add-fennel-path libs/?.fnl --add-package-path libs/?.lua
and both libraries can be used without any clash:
(local http (require "io.gitlab.andreyorst.fnl-http"))
(local async (require "io.gitlab.andreyorst.async"))
(local some-channel (async.chan))
;; ----8<----
(http.post "some.url" {:body some-channel})
---
If Fennel had an official package manager, I think this wouldn't be a
big issue. I have two more libraries that have to be shared in this
way:
- reduced.lua[3]
- lazy-seq[4]
Both implement some kind of singleton for integration purposes and fast
checks. The biggest library of mine - fennel-cljlib[5] features an
install script that installs the bundled versions of libraries alongside
the library itself. It significantly simplifies the building process,
compared to the manual textual patching I had to do for each release.
Having an official package manager would solve this because dependencies
would be installed in a predictable manner. Without it, however, I
figured that a reverse domain naming scheme would provide sufficient
enough protection because probably nobody would create another library
that would get installed in the same place as mine.
Let me know what you think.
[1] https://gitlab.com/andreyorst/fnl-http
[2] https://gitlab.com/andreyorst/async.fnl
[3] https://gitlab.com/andreyorst/reduced.lua
[4] https://gitlab.com/andreyorst/lazy-seq
[5] https://gitlab.com/andreyorst/fennel-cljlib
--
Andrey Listopadov
Andrey Listopadov <andreyorst@gmail.com> writes:
> For some time I've been working on a library for making HTTP requests[1]
> asynchronously from Fennel without relying on anything but luasocket. I
> have a library for asynchronous programming async.fnl[2] that implements
> a channel-based workflow, and uses debug.sethook for scheduling. I've
> used this library in fnl-http, and faced a certain problem - how do I
> bundle async.fnl in such a way that fnl-http can still be a single file
> library for ease of use while making sure that users can still use
> async.fnl without being affected by the bundled version?
This is a really interesting post that I think deserves more thought
than I am able to dedicate to it right now! I hope to write up a reply
at some point after the game jam is over.
Thanks for your patience.
-Phil
Phil Hagelberg <phil@hagelb.org> writes:
> [[PGP Signed Part:Undecided]]
> Andrey Listopadov <andreyorst@gmail.com> writes:
>
>> For some time I've been working on a library for making HTTP requests[1]
>> asynchronously from Fennel without relying on anything but luasocket. I
>> have a library for asynchronous programming async.fnl[2] that implements
>> a channel-based workflow, and uses debug.sethook for scheduling. I've
>> used this library in fnl-http, and faced a certain problem - how do I
>> bundle async.fnl in such a way that fnl-http can still be a single file
>> library for ease of use while making sure that users can still use
>> async.fnl without being affected by the bundled version?
>
> This is a really interesting post that I think deserves more thought
> than I am able to dedicate to it right now! I hope to write up a reply
> at some point after the game jam is over.
>
> Thanks for your patience.
>
> -Phil
>
> [[End of PGP Signed Part]]
So, after discussing this at the User group meeting I threw together a
prototype of a package manager (this is maybe the fifth time I'm writing
this kind of script):
#!/bin/env fennel
;; -*- mode: fennel -*-
(local fennel
(require :fennel))
(local deps-dir
(.. (os.getenv "PWD") "/" ".deps"))
(fn read-deps [file]
(fennel.dofile file {:env {}}))
(fn luarocks-available? []
(pick-values 1
(os.execute "luarocks --help >/dev/null 2>&1")))
(fn rock-installed? [rock]
(pick-values 1
(os.execute (.. "luarocks --tree " deps-dir " show " rock " >/dev/null 2>&1"))))
(fn process-rock [name {: version}]
(assert (luarocks-available?) "can't access luarocks executable")
(when (not (rock-installed? name))
(print (.. "processing rock: " name))
(let [out (os.tmpname)
command (string.format
"(luarocks --tree %s install %s %s)>%s 2>&1"
deps-dir
name
(or version "")
out)]
(assert
(os.execute command)
(.. "can't process " (fennel.view name) ". Log: " out)))))
(fn luarocks-path [flag]
(assert (luarocks-available?) "can't access luarocks executable")
(with-open [p (io.popen (.. "luarocks --tree " deps-dir " path " (or flag "") " --full 2>/dev/null"))]
(icollect [path (: (p:read :*l) :gmatch "([^;]+)")] path)))
(fn url-from-dep [name]
(let [(reverse-domain project)
(name:match "([^/]+)/(.-)$")
[_ domain &as components]
(if (and reverse-domain project)
(icollect [dep (reverse-domain:gmatch "([^.]+)%.?")] dep)
[])]
(case domain
(where (or :github :gitlab))
(let [[_ _ & components] components]
(string.format "https://%s.com/%s/%s" domain (table.concat components "/") project))
:sr
(let [[_ _ & components] components]
(string.format "https://git.sr.ht/%s/%s" (table.concat components "/") project))
_ (error (.. "uable to determine URL for: " (fennel.view name)) 2))))
(fn file-exists? [filename]
(case (io.open filename :r)
f (do (f:close) true)
_ false))
(fn process-git [name {: tag : url} paths all-paths process-deps]
(let [out (os.tmpname)
url (or url (url-from-dep name))
name (url:match ".*/(.-)$")]
(when (not (file-exists? (.. deps-dir "/" name)))
(print (.. "processing git repo: " url))
(assert
(os.execute
(.. "(cd " deps-dir
" && git clone" (if tag (.. " --branch " tag " ") " ")
url " " name
") > " out " 2>&1"))
(.. "can't process " (fennel.view name) ". Log: " out)))
(process-deps (.. deps-dir "/" name "/" "deps.fnl") all-paths)))
(fn process-deps [deps-file all-paths]
(when (file-exists? deps-file)
(let [base-dir (or (deps-file:match "^(.+)/.*$") "./")
{: deps : paths} (read-deps deps-file)]
(each [variable paths (pairs (or paths {}))]
(tset all-paths variable
(icollect [_ p (ipairs paths)
:into (or (. all-paths variable) [])]
(.. base-dir "/" p))))
(os.execute (.. "mkdir -p " deps-dir))
(each [dep {:type t &as descr} (pairs (or deps {}))]
(case t
:git (process-git dep descr (or paths {}) all-paths process-deps)
:rock (process-rock dep descr)
_ (error (.. "unsupported dependency type: " t) 2))))))
(fn process-paths [paths all-paths]
(accumulate [res (or all-paths {})
variable paths (pairs paths)]
(let [variable
(case variable
:fennel :fennel-path
:lua :package-path
:clua :package-cpath
:macro :macro-path
_ (error (.. "unsupported path type:" (fennel.view variable)) 2))]
(doto res
(tset variable
(icollect [_ p (ipairs paths)
:into (or (. res variable) [])]
p))))))
(fn main []
(let [all-paths {}]
(case (pcall process-deps (.. (os.getenv "PWD") "/" "deps.fnl") all-paths)
(false msg)
(do (io.stderr:write msg "\n")
(os.exit 1))
_ (->> (process-paths all-paths)
(process-paths {:lua (luarocks-path "--lr-path")
:clua (luarocks-path "--lr-cpath")})))))
(let [paths (main)]
(when paths.fennel-path
(set fennel.path
(table.concat (doto paths.fennel-path
(table.insert fennel.path))
";")))
(when paths.package-path
(set package.path
(table.concat (doto paths.package-path
(table.insert package.path))
";")))
(when paths.package-cpath
(set package.cpath
(table.concat (doto paths.package-cpath
(table.insert package.cpath))
";")))
(when paths.macro-path
(set fennel.macro-path
(table.concat (doto paths.macro-path
(table.insert fennel.macro-path))
";"))))
(fennel.repl)
This is as simple as it gets, there's no proper support for Windows,
etc, and mostly only happy paths were coded.
With this script in my PATH I've set up some `deps.fnl` files in my
projects:
The fnl-http project now has the following deps.fnl:
{:deps
{:async.fnl {:type "git"
:url "https://gitlab.com/andreyorst/async.fnl"
:tag "main"}
:io.gitlab.andreyorst/json.fnl {:type "git" :tag "main"}
:io.gitlab.andreyorst/reader.fnl {:type "git" :tag "main"}
:luasocket {:type "rock" :version nil}}
:paths {:fennel ["src/?.fnl" "src/?/init.fnl"]}}
The json.fnl has the following deps.fnl:
{:paths {:fennel ["src/?.fnl"]}
:deps {:io.gitlab.andreyorst/reader.fnl {:type "git" :tag "main"}}}
And async.fnl, and reader.fnl have the following deps.fnl in each project:
{:paths {:fennel ["src/?.fnl"]}}
After calling the script from above in the root directory of the
fnl-http project, the `.deps` directory is created with the following
structure:
.deps
├── async.fnl
│ ├── deps.fnl
│ ├── ...
│ └── src
│ └── io
│ └── gitlab
│ └── andreyorst
│ └── async.fnl
├── json.fnl
│ ├── deps.fnl
│ ├── ...
│ └── src
│ └── io
│ └── gitlab
│ └── andreyorst
│ └── json.fnl
├── lib
│ ├── lua
│ │ └── 5.4
│ │ └── socket
│ │ ├── core.so
│ │ ├── serial.so
│ │ └── unix.so
│ └── luarocks
│ └── rocks-5.4
│ ├── luasocket
│ │ └── ...
│ └── manifest
├── reader.fnl
│ ├── deps.fnl
│ └── src
│ └── io
│ └── gitlab
│ └── andreyorst
│ └── reader.fnl
└── share
└── lua
└── 5.4
├── ltn12.lua
├── socket
│ ├── ftp.lua
│ ├── headers.lua
│ ├── http.lua
│ ├── smtp.lua
│ ├── tp.lua
│ └── url.lua
└── socket.lua
The script collects all relevant paths for each dependency, and sets up
the REPL with these paths configured.
The idea here is again automation of where libraries are installed, and
how paths are specified.
This is not pushed anywhere yet, but I want to hear some thoughts and
get feedback. I'll set separate branches in these projects and push
this later this week, so others could try it if necessary.
--
Andrey Listopadov
> This is not pushed anywhere yet, but I want to hear some thoughts and
> get feedback. I'll set separate branches in these projects and push
> this later this week, so others could try it if necessary.
I've pushed a bit improved version of the script to:
https://gitlab.com/andreyorst/deps.fnl
Should be a bit more robust on Windows machines, and such.
Still, not extensively tested, but can be tried with my fnl-http project
on the deps branch:
https://gitlab.com/andreyorst/fnl-http/-/tree/deps
After a bit of refactoring, this script now acts as a wrapper around the
fennel script. So it can be pretty much included in the launcher.fnl.
Here's how it looks when I launch deps in the fnl-http project on the
deps branch:
$ deps --repl
processing git repo: https://gitlab.com/andreyorst/async.fnl
processing git repo: https://gitlab.com/andreyorst/json.fnl
processing git repo: https://gitlab.com/andreyorst/reader.fnl
processing git repo: https://gitlab.com/andreyorst/reader.fnl
processing rock: luasocket
Welcome to Fennel 1.5.1-dev on PUC Lua 5.4!
Use ,help to see available commands.
>> (local http (require :io.gitlab.andreyorst.fnl-http))
nil
>> (http.get "httpbin.org/get" {:as :json})
{:body {:args {}
:headers {:Host "httpbin.org"
:X-Amzn-Trace-Id "Root=1-673f7cf7-6d03d6c4478a7d4423edcccc"}
:origin "185.9.75.213"
:url "http://httpbin.org/get"}
:headers {:Access-Control-Allow-Credentials "true"
:Access-Control-Allow-Origin "*"
:Connection "keep-alive"
:Content-Length "197"
:Content-Type "application/json"
:Date "Thu, 21 Nov 2024 18:33:27 GMT"
:Server "gunicorn/19.9.0"}
:http-client #<tcp-client: 0x559389a08af0>
:length 197
:protocol-version {:major 1 :minor 1 :name "HTTP"}
:reason-phrase "OK"
:request-time 221
:status 200
:trace-redirects {}}
>>
So it works as one would expect.
I can also do this:
deps --require-as-include -c src/io/gitlab/andreyorst/fnl-http/init.fnl
And it will correctly compile everything into a self-contained Lua file.
However, an actual command generated and executed by the deps script looks like this:
fennel --add-fennel-path /var/home/alist/Projects/Fennel/fnl-http/src/?.fnl \
--add-fennel-path /var/home/alist/Projects/Fennel/fnl-http/src/?/init.fnl \
--add-fennel-path /var/home/alist/Projects/Fennel/fnl-http/.deps/git/async.fnl/deps/async.fnl/src/?.fnl \
--add-fennel-path /var/home/alist/Projects/Fennel/fnl-http/.deps/git/json.fnl/deps/json.fnl/src/?.fnl \
--add-fennel-path /var/home/alist/Projects/Fennel/fnl-http/.deps/git/reader.fnl/deps/reader.fnl/src/?.fnl \
--add-package-path /var/home/alist/Projects/Fennel/fnl-http/.deps/rocks/share/lua/5.4/?.lua \
--add-package-path /var/home/alist/Projects/Fennel/fnl-http/.deps/rocks/share/lua/5.4/?/init.lua \
--add-package-path /var/home/alist/.luarocks/share/lua/5.4/?.lua \
--add-package-path /var/home/alist/.luarocks/share/lua/5.4/?/init.lua \
--add-package-path /var/home/alist/.local/share/lua/5.4/?.lua \
--add-package-path /var/home/alist/.local/share/lua/5.4/?/init.lua \
--add-package-path /usr/share/lua/5.4/?.lua \
--add-package-path /usr/share/lua/5.4/?/init.lua \
--add-package-path /usr/lib64/lua/5.4/?.lua \
--add-package-path /usr/lib64/lua/5.4/?/init.lua \
--add-package-path ./?.lua \
--add-package-path ./?/init.lua \
--add-package-cpath /var/home/alist/Projects/Fennel/fnl-http/.deps/rocks/lib/lua/5.4/?.so \
--add-package-cpath /var/home/alist/.luarocks/lib/lua/5.4/?.so \
--add-package-cpath /var/home/alist/.local/lib/lua/5.4/?.so \
--add-package-cpath /usr/lib64/lua/5.4/?.so \
--add-package-cpath /usr/lib64/lua/5.4/loadall.so \
--add-package-cpath ./?.so \
--require-as-include \
-c src/io/gitlab/andreyorst/fnl-http/init.fnl
(I've formatted it a bit for readability)
All these flags were constructed based on the information from deps.fnl
files stored in each project used as a dependency.
--
Andrey Listopadov