~technomancy/fennel

fennel: allow require-as-include to work with non-literal include paths v3 APPLIED

Andrey Listopadov: 1
 allow require-as-include to work with non-literal include paths

 11 files changed, 242 insertions(+), 28 deletions(-)
#555948 .build.yml success
Export patchset (mbox)
How do I use this?

Copy & paste the following snippet into your terminal to import this patchset into git:

curl -s https://lists.sr.ht/~technomancy/fennel/patches/24090/mbox | git am -3
Learn more about email & git
View this thread in the archives

[PATCH fennel v3] allow require-as-include to work with non-literal include paths Export this patch

This change allows library writers to create libraries with nested
modules, which can be required relatively to the root of the library,
making it possible to then build a self contained application, with
all library modules included into the source, regardless of how
library directory is named or the path to the library.
---
 changelog.md             |   2 +
 reference.md             |  20 ++++--
 src/fennel/specials.fnl  |  49 ++++++++-----
 test/misc.fnl            |  25 ++++++-
 test/mod/foo3.fnl        |   5 ++
 test/mod/foo4.fnl        |   5 ++
 test/mod/foo5.fnl        |   5 ++
 test/mod/foo6.fnl        |   5 ++
 test/mod/nested/mod1.fnl |   1 +
 test/mod/nested/mod2.fnl |   4 ++
 tutorial.md              | 149 ++++++++++++++++++++++++++++++++++++++-
 11 files changed, 242 insertions(+), 28 deletions(-)
 create mode 100644 test/mod/foo3.fnl
 create mode 100644 test/mod/foo4.fnl
 create mode 100644 test/mod/foo5.fnl
 create mode 100644 test/mod/foo6.fnl
 create mode 100644 test/mod/nested/mod1.fnl
 create mode 100644 test/mod/nested/mod2.fnl

diff --git a/changelog.md b/changelog.md
index 4091d44..cd35f33 100644
--- a/changelog.md
+++ b/changelog.md
@@ -4,6 +4,8 @@ Changes are **marked in bold** which could result in backwards-incompatibility.

## 0.10.0 / ???

* Allow using expressions in `include` and make `--require-as-include`
  resolve module names dynamically.  See the require section in the reference.
* Add `,apropos pattern` and `,apropos-doc pattern` repl commands
* Deprecate `pick-args` macro
* Support repl completion on methods inside tables
diff --git a/reference.md b/reference.md
index fd25267..bc1cb9e 100644
--- a/reference.md
+++ b/reference.md
@@ -975,10 +975,10 @@ subsequent forms are evaluated solely for side-effects.
(include :my.embedded.module)
```

Loads Fennel/Lua module code at compile time and embeds it in the compiled
output. The module name must be a string literal that can resolve to
a module during compilation.  The bundled code will be wrapped in a
function invocation in the emitted Lua and set on
Loads Fennel/Lua module code at compile time and embeds it in the
compiled output. The module name must resolve to a string literal
during compilation.  The bundled code will be wrapped in a function
invocation in the emitted Lua and set on
`package.preload[modulename]`; a normal `require` is then emitted
where `include` was used to load it on demand as a normal module.

@@ -987,9 +987,15 @@ In most cases it's better to use `require` in your code and use the
`--require-as-include` CLI flag (`fennel --help`) to accomplish this.

The `require` function is not part of Fennel; it comes from
Lua. However, it works to load Fennel code. See the end of
[the tutorial](tutorial.md) and [Programming in Lua][5] for details
about `require`.
Lua. However, it works to load Fennel code. See the [Modules and
multiple files](tutorial#modules-and-multiple-files) section in the
tutorial and [Programming in Lua][5] for details about `require`.

Starting from version 0.10.0 `include` and hence
`--require-as-include` support semi-dynamic compile-time resolution of
module paths similarly to `import-macros`.  See the [relative
require](tutorial#relative-require) section in the tutorial for more
information.

## Macros

diff --git a/src/fennel/specials.fnl b/src/fennel/specials.fnl
index 8d74bd9..4dc503e 100644
--- a/src/fennel/specials.fnl
+++ b/src/fennel/specials.fnl
@@ -1157,15 +1157,26 @@ modules in the compiler environment."
                     "expected each macro to be function" ast)
    (tset scope.macros k v)))

(fn resolve-module-name [ast scope parent opts]
  ;; Compile module path to resolve real module name.  Allows using
  ;; (.. ... :.foo.bar) expressions and self-contained
  ;; statement-expressions in `require`, `include`, `require-macros`,
  ;; and `import-macros`.
  (let [filename (or ast.filename (. ast 2 :filename))
        module-name utils.root.options.module-name
        modexpr (compiler.compile (. ast 2) opts)
        modname-chunk (load-code modexpr)]
    (match (pcall modname-chunk module-name filename)
      (true modname) (utils.expr (string.format "%q" modname) :literal)
      _ (. (compiler.compile1 (. ast 2) scope parent {:nval 1}) 1))))

(fn SPECIALS.require-macros [ast scope parent real-ast]
  (compiler.assert (= (length ast) 2) "Expected one module name argument"
                   (or real-ast ast)) ; real-ast comes from import-macros
  ;; don't require modname to be string literal; it just needs to compile to one
  (let [filename (or (. ast 2 :filename) ast.filename)
        modname-chunk (load-code (compiler.compile (. ast 2)) nil filename)
        modname (modname-chunk utils.root.options.module-name filename)]
    (compiler.assert (= (type modname) :string)
                     "module name must compile to string" (or real-ast ast))
  (let [modexpr (resolve-module-name ast scope parent {})
        _ (compiler.assert (= modexpr.type :literal)
                           "module name must compile to string" (or real-ast ast))
        modname ((load-code (.. "return " (. modexpr 1))))]
    (when (not (. macro-loaded modname))
      (let [env (make-compiler-env ast scope parent)
            (loader filename) (search-macro-module modname 1)]
@@ -1230,21 +1241,25 @@ Consider using import-macros instead as it is more flexible.")

(fn SPECIALS.include [ast scope parent opts]
  (compiler.assert (= (length ast) 2) "expected one argument" ast)
  (let [modexpr (. (compiler.compile1 (. ast 2) scope parent {:nval 1}) 1)]
  (let [modexpr (resolve-module-name ast scope parent opts)]
    (if (or (not= modexpr.type :literal) (not= (: (. modexpr 1) :byte) 34))
        (if opts.fallback
            (opts.fallback modexpr)
            (compiler.assert false "module name must be string literal" ast))
        (let [mod ((load-code (.. "return " (. modexpr 1))))]
          (or (include-circular-fallback mod modexpr opts.fallback ast)
              (. utils.root.scope.includes mod) ; check cache
              ;; Find path to Fennel or Lua source; prefering Fennel
              (match (search-module mod)
                fennel-path (include-path ast opts fennel-path mod true)
                _ (let [lua-path (search-module mod package.path)]
                    (if lua-path (include-path ast opts lua-path mod false)
                        opts.fallback (opts.fallback modexpr)
                        (compiler.assert false (.. "module not found " mod) ast)))))))))
        (let [mod ((load-code (.. "return " (. modexpr 1))))
              oldmod utils.root.options.module-name
              _ (set utils.root.options.module-name mod)
              res (or (include-circular-fallback mod modexpr opts.fallback ast)
                      (. utils.root.scope.includes mod) ; check cache
                      ;; Find path to Fennel or Lua source; prefering Fennel
                      (match (search-module mod)
                        fennel-path (include-path ast opts fennel-path mod true)
                        _ (let [lua-path (search-module mod package.path)]
                            (if lua-path (include-path ast opts lua-path mod false)
                                opts.fallback (opts.fallback modexpr)
                                (compiler.assert false (.. "module not found " mod) ast)))))]
          (set utils.root.options.module-name oldmod)
          res))))

(doc-special :include [:module-name-literal]
             "Like require but load the target module during compilation and embed it in the
diff --git a/test/misc.fnl b/test/misc.fnl
index 33a2241..45d702c 100644
--- a/test/misc.fnl
+++ b/test/misc.fnl
@@ -26,15 +26,36 @@
  (let [expected "foo:FOO-1bar:BAR-2-BAZ-3"
        (ok out) (pcall fennel.dofile "test/mod/foo.fnl")
        (ok2 out2) (pcall fennel.dofile "test/mod/foo2.fnl"
                          {:requireAsIncluede true})]
    (l.assertTrue ok "Expected foo to run")
                          {:requireAsInclude true})
        (ok3 out3) (pcall fennel.dofile "test/mod/foo3.fnl"
                          {:requireAsInclude true})
        (ok4 out4) (pcall fennel.dofile "test/mod/foo4.fnl")
        (ok5 out5) (pcall fennel.dofile "test/mod/foo5.fnl"
                          {:requireAsInclude true}
                          :test)
        (ok6 out6) (pcall fennel.dofile "test/mod/foo6.fnl"
                          {:requireAsInclude true}
                          :test)]
    (l.assertTrue ok out)
    (l.assertTrue ok2 "Expected foo2 to run")
    (l.assertTrue ok3 "Expected foo3 to run")
    (l.assertTrue ok4 "Expected foo4 to run")
    (l.assertTrue ok5 "Expected foo5 to run")
    (l.assertTrue ok6 "Expected foo6 to run")
    (l.assertEquals (and (= :table (type out)) out.result) expected
                    (.. "Expected include to have result: " expected))
    (l.assertFalse out.quux
                   "Expected include not to leak upvalues into included modules")
    (l.assertEquals (view out) (view out2)
                    "Expected requireAsInclude to behave the same as include")
    (l.assertEquals (view out) (view out3)
                    "Expected requireAsInclude to behave the same as include when given an expression")
    (l.assertEquals (view out) (view out4)
                    "Expected include to work when given an expression")
    (l.assertEquals (view out) (view out5)
                    "Expected relative requireAsInclude to work when given a ...")
    (l.assertEquals (view out) (view out6)
                    "Expected relative requireAsInclude to work with nested modules")
    (l.assertNil _G.quux "Expected include to actually be local")
    (let [spliceOk (pcall fennel.dofile "test/mod/splice.fnl")]
      (l.assertTrue spliceOk "Expected splice to run")
diff --git a/test/mod/foo3.fnl b/test/mod/foo3.fnl
new file mode 100644
index 0000000..d955297
--- /dev/null
+++ b/test/mod/foo3.fnl
@@ -0,0 +1,5 @@
(local foo [:FOO 1])
(local quux (require (.. :test :.mod.quux)))
(local bar (require (.. :test :.mod :.bar)))
{:result (.. "foo:" (table.concat foo "-") "bar:" (table.concat bar "-"))
 : quux}
diff --git a/test/mod/foo4.fnl b/test/mod/foo4.fnl
new file mode 100644
index 0000000..c4f62e5
--- /dev/null
+++ b/test/mod/foo4.fnl
@@ -0,0 +1,5 @@
(local foo [:FOO 1])
(local quux (include (.. :test :.mod.quux)))
(local bar (include (.. :test :.mod :.bar)))
{:result (.. "foo:" (table.concat foo "-") "bar:" (table.concat bar "-"))
 : quux}
diff --git a/test/mod/foo5.fnl b/test/mod/foo5.fnl
new file mode 100644
index 0000000..2db4b56
--- /dev/null
+++ b/test/mod/foo5.fnl
@@ -0,0 +1,5 @@
(local foo [:FOO 1])
(local quux (require (.. ... :.mod.quux)))
(local bar (require (.. ... :.mod :.bar)))
{:result (.. "foo:" (table.concat foo "-") "bar:" (table.concat bar "-"))
 : quux}
diff --git a/test/mod/foo6.fnl b/test/mod/foo6.fnl
new file mode 100644
index 0000000..3f5666e
--- /dev/null
+++ b/test/mod/foo6.fnl
@@ -0,0 +1,5 @@
(local foo [:FOO 1])
(local quux (require (.. ... :.mod.quux)))
(local bar (require (.. ... :.mod.nested.mod1)))
{:result (.. "foo:" (table.concat foo "-") "bar:" (table.concat bar "-"))
 : quux}
diff --git a/test/mod/nested/mod1.fnl b/test/mod/nested/mod1.fnl
new file mode 100644
index 0000000..c5b7070
--- /dev/null
+++ b/test/mod/nested/mod1.fnl
@@ -0,0 +1 @@
(require (: (or ... "") :gsub "(nested%.).*$" "%1mod2"))
diff --git a/test/mod/nested/mod2.fnl b/test/mod/nested/mod2.fnl
new file mode 100644
index 0000000..c43df90
--- /dev/null
+++ b/test/mod/nested/mod2.fnl
@@ -0,0 +1,4 @@
(local bar [:BAR 2])
(each [_ v (ipairs (include :test.mod.baz))]
  (table.insert bar v))
bar
diff --git a/tutorial.md b/tutorial.md
index 1ed66d9..33576ce 100644
--- a/tutorial.md
+++ b/tutorial.md
@@ -538,11 +538,11 @@ use `_G.myglobal` to refer to it in a way that works around this check.

Another possible cause for this error is a modified [function environment][16].
The solution depends on how you're using Fennel:
* Embedded Fennel can have its searcher modified to ignore certain (or all) 
* Embedded Fennel can have its searcher modified to ignore certain (or all)
  globals via the `allowedGlobals` parameter. See the [Lua API][17] page for
  instructions.
* Fennel's CLI has the `--globals` parameter, which accepts a comma-separated
  list of globals to ignore. For example, to disable strict mode for globals 
  list of globals to ignore. For example, to disable strict mode for globals
  x, y, and z:
  ```shell
  fennel --globals x,y,z yourfennelscript.fnl
@@ -627,6 +627,151 @@ which is distinct from `package.path` used to find Lua modules). The
path usually includes an entry to let you load things relative to the
current directory by default.

## Relative require

There are several ways of how to write a library which uses nested modules.
One of which is to rely on something like Luarocks, to manage library installation and availability of it and it's modules.
Another way is using relative require style for loading nested modules.
With relative require, libraries don't depend on the root directory name or its location when resolving inner module paths.

For example, here's small a library, which contains `init.fnl` file at the root directory, and a subdirectory `utils` with a file `private.fnl` that contains private library API.
Such libraries are usually written like this:

```fennel
;; file example/init.fnl:
(local p (require :example.utils.private))
(fn publicfn [] (p.privatefn))
{:publicfn publicfn}
```

for the main module.
Which requires `example.utils.private` module:

```fennel
;; file example/utils/private.fnl:
(local secret "super-secret-thing")
(fn privatefn [] secret)
{:privatefn privatefn}
```

One issue here is that the path to the library must be exactly `example`, e.g. `(require :example)`, which can't be enforced on the library user.
For example, the library can be put into `libs` directory of the project to avoid the top level cluttering, and required as `(require :libs.example)`.
This will not work, because library itself will try to require `:example.utils.private` and not `:libs.example.utils.private`, which is now a correct module path.
Another example of breakage can happen when library is pulled from git, and the directory was renamed by the user like this: `git clone /url/to/example.git example-lib`.

Luarocks addresses this problem by enforcing both the directory name, and installation path, populating `LUA_PATH` environment variable to make the library available.
This, of course, can be done manually by setting `LUA_PATH` per project in the build pipeline, pointing it to the right directory.
But it's not very transparent, and when requiring some libraries it's better to see full path, that directly maps to the project's file structure rather than looking up where the `LUA_PATH` is modified.

In Fennel ecosystem we encourage a simpler way of managing libraries in the project.
Simply dropping a library into your project's tree or using git submodule is usually enough, and the require paths should be handled by the library itself.

Here's how a relative require path can be specified in the `example/init.fnl` to make this library name/path agnostic:

```fennel
;;; file example/init.fnl:
(local p (require (.. ... :.utils.private)))
(fn publicfn [] (p.privatefn))
{: publicfn}
```

This way, when requiring the library with `(require :lib.example)`, the first value in `...` will hold the `"lib.example"` string.
This string is then concatenated with the `".utils.private"`, and `require` will properly find and load the nested module at runtime under the `"lib.example.utils.private"` path.
This is a Lua feature, and not something Fennel specific, and it will work the same when library is AOT compiled to Lua.

Since Fennel version 0.10.0 this also works at compile-time, when using `include` special or the `--require-as-include` flag, with the constraint that the expression can be computed at compile time.
Compile time here means that the expression must be self-contained, e.g. don't refer to locals or globals, but embed all values directly.
In other words, this will only work at runtime, but not with `include` or `--require-as-include`:

```fennel
(local current-module ...)
(require (.. current-module :.nested-module))
```

This won't work with `include` because `current-module` is not known at compile time.
This, on the other hand, will work both at runtime and at compile time:

```fennel
(require (.. ... :.nested-module))
```

It works because the `...` args are propagated during compilation, so when the application, which uses this library is compiled, all library code is correctly included into the self-contained Lua file.
For a better illustration of the idea, here's the example project structure, with all the code from the above, plus `main.fnl` file which uses the example library.

```
$ tree .
├ libs
│ └ example
│   ├ init.fnl
│   └ utils
│     └ private.fnl
└ lib-user.fnl
$ cat main.fnl
(local example (require :libs.example))
(print (example.publicfn))
$ fennel --require-as-include --compile main.fnl
```

Compiling `main.fnl` with the `--require-as-include` flag produces the following Lua code:

```lua
local example
package.preload["libs.example.utils.private"] = package.preload["libs.example.utils.private"] or function(...)
  local function privatefn()
    return "super-secret-thing"
  end
  return {privatefn = privatefn}
end
package.preload["libs.example"] = package.preload["libs.example"] or function(...)
  local p = require("libs.example.utils.private")
  local function publicfn()
    return p.privatefn()
  end
  return {publicfn = publicfn}
end
example = require("libs.example")
return print(example.publicfn())
```

Note that `package.preload` entries contain fully qualified paths like `"libs.example.utils.private"`, which were resolved at compile time.

### Multiple nested modules

When the library uses nested modules, that reference other modules, it gets a bit trickier.
To require a nested module from another nested module we must keep the prefix up to current module, but remove the module name itself.
For example, let's add another module to `libs/example/utils/another-mod.fnl` and reference it from `libs/example/utils/private.fnl`:

```fennel
;; file example/utils/another-mod.fnl:
(fn inc [x] (+ x 1))
```

And we can reference this module like this:

```fennel
;; file example/utils/private.fnl:
(local inc (require (string.gsub ... "(utils%.)private$" "%1another-mod")))
(local secret "super-secret-thing")
(fn anotherfn [x] (inc x))
(fn privatefn [] secret)
{: privatefn
 : anotherfn}
```

Compiling the `main.fnl` with `--require-as-include` will add the following block of code to the output file, that references the module:

```lua
package.preload["libs.example.utils.another-mod"] = package.preload["libs.example.utils.another-mod"] or function(...)
  local function inc(x)
    return (x + 1)
  end
  return {inc = inc}
end
-- rest of compiler output is the same as in previous section
```

Other modules will use simple way to reference other modules, with the only difference being their name in the pattern.

[1]: https://stopa.io/post/265
[2]: http://danmidwood.com/content/2014/11/21/animated-paredit.html
[3]: https://www.lua.org/manual/5.1/
-- 
2.31.1
Andrey Listopadov <andreyorst@gmail.com> writes: