Add support for fennel.macro-searchers. v1 SUPERSEDED

Phil Hagelberg: 1
 Add support for fennel.macro-searchers.

 4 files changed, 58 insertions(+), 22 deletions(-)
Export patchset (mbox)
How do I use this?

Copy & paste the following snippet into your terminal to import this patchset into git:

curl -s https://lists.sr.ht/~technomancy/fennel/patches/21778/mbox | git am -3
Learn more about email & git
View this thread in the archives

[PATCH] Add support for fennel.macro-searchers. Export this patch

We now have a fennel.macro-searchers table which functions as a
compile-time equivalent to the package.searchers table; anyone can
insert their own searcher functions to change how macros can be found.

The main thing I don't like about this is that it exposes our little
internal trick of passing :env :_COMPILER in the options table to load
things in compiler scope. We have supported this for the environment
for internal reasons for a while, but now we both expand it (to also
affect :scope) and document it as part of the interface, in order to
simplify the implementation of the default macro searcher function.

I don't love that trick; it was originally added in order to allow us
to load plugins in compiler scope without reaching into the guts of
the compiler module, but it feels like a bit of a hack.

We could maybe make it cleaner by exposing make-compiler-scope and
make-compiler-env as part of the public API, but I'm not sure it's
necessary. If this shorthand gets the same job done as adding two more
functions to our already-big API, maybe that's fine.
 changelog.md            |  1 +
 reference.md            | 32 +++++++++++++++++++++++++++++-
 src/fennel.fnl          |  3 +++
 src/fennel/specials.fnl | 44 +++++++++++++++++++++--------------------
 4 files changed, 58 insertions(+), 22 deletions(-)

diff --git a/changelog.md b/changelog.md
index 542491a..0fbf67d 100644
--- a/changelog.md
+++ b/changelog.md
@@ -2,6 +2,7 @@

## 0.9.0 / ???

* Add `macro-searchers` table for finding macros similarly to `package.searchers`
* Support `&as` inside pattern matches
* Include stack trace for errors during macroexpansion
* The `sym` function in compile scope now takes a source table second argument
diff --git a/reference.md b/reference.md
index 4615634..9b1f071 100644
--- a/reference.md
+++ b/reference.md
@@ -1049,7 +1049,37 @@ inside compiler scope which macros run in.

The `require-macros` form is like `import-macros`, except it does not
give you any control over the naming of the macros being
imported. Consider using `import-macros` instead of `require-macros`.
imported. It is strongly recommended to use `import-macros` instead.

### Macro module searching

By default, Fennel will search for macro modules using the same logic
it uses to search for normal runtime modules: by walking thru entries
on `fennel.path` and checking the filesystem for matches. However, in
some cases this might not be suitable, for instance if your Fennel
program is packaged in some kind of archive file and the modules do
not exist as distinct files on disk.

To support this case you can add your own searcher function to the
`fennel.macro-searchers` table. For example, assuming `find-in-archive`
is a function which can look up strings from the archive given a path:

(local fennel (require :fennel))

(fn my-searcher [module-name]
  (let [filename (.. "src/" module-name)]
    (match (find-in-archive filename)
      code (values (partial fennel.eval code {:env :_COMPILER})

(table.insert fennel.macro-searchers my-searcher)

The searcher function should take a module name as a string and return
two values if it can find the macro module: a loader function which will
return the macro table when called, and an optional filename. The
loader function will receive the module name and the filename as arguments.

### `macros` define several macros

diff --git a/src/fennel.fnl b/src/fennel.fnl
index 5615708..80aa2e1 100644
--- a/src/fennel.fnl
+++ b/src/fennel.fnl
@@ -50,6 +50,8 @@
    ;; to provide targeted error messages.
    (when (and (not opts.filename) (not opts.source))
      (set opts.source str))
    (when (= opts.env :_COMPILER)
      (set opts.scope (compiler.make-scope compiler.scopes.compiler)))

(fn eval [str options ...]
@@ -98,6 +100,7 @@
            :gensym compiler.gensym
            :load-code specials.load-code
            :macro-loaded specials.macro-loaded
            :macro-searchers specials.macro-searchers
            :search-module specials.search-module
            :make-searcher specials.make-searcher
            :makeSearcher specials.make-searcher
diff --git a/src/fennel/specials.fnl b/src/fennel/specials.fnl
index 1731ada..628690f 100644
--- a/src/fennel/specials.fnl
+++ b/src/fennel/specials.fnl
@@ -1044,17 +1044,18 @@ table.insert(package.loaders, fennel.searcher)"
      (table.insert allowed k))

(fn compiler-env-domodule [modname env ?ast ?scope]
  (let [filename (compiler.assert (search-module modname)
                                  (.. modname " module not found.") ?ast)
        globals (macro-globals env (current-global-names))
        scope (or ?scope (compiler.make-scope compiler.scopes.compiler))]
    (utils.fennel-module.dofile filename
                                {:allowedGlobals globals
                                 :useMetadata utils.root.options.useMetadata
                                 : env
                                 : scope}
                                modname filename)))
(fn default-macro-searcher [module-name]
  (match (search-module module-name)
    filename (values (partial utils.fennel-module.dofile filename
                              {:env :_COMPILER}) filename)))

(local macro-searchers [default-macro-searcher])

(fn search-macro-module [modname n]
  (match (. macro-searchers n)
    f (match (f modname)
        (loader filename) (values loader filename)
        _ (search-macro-module modname (+ n 1)))))

;; This is the compile-env equivalent of package.loaded. It's used by
;; require-macros and import-macros, but also by require when used from within
@@ -1074,11 +1075,10 @@ table.insert(package.loaders, fennel.searcher)"
It ensures that compile-scoped modules are loaded differently from regular
modules in the compiler environment."
                    (or (. macro-loaded modname) (metadata-only-fennel modname)
                        (let [scope (compiler.make-scope compiler.scopes.compiler)
                              env (make-compiler-env nil scope nil)
                              mod (compiler-env-domodule modname env nil scope)]
                          (tset macro-loaded modname mod)
                        (let [(loader filename) (search-macro-module modname 1)]
                          (compiler.assert loader (.. modname " module not found."))
                          (tset macro-loaded modname (loader modname filename))
                          (. macro-loaded modname)))))

(fn add-macros [macros* ast scope]
  (compiler.assert (utils.table? macros*) "expected macros to be table" ast)
@@ -1092,14 +1092,15 @@ modules in the compiler environment."
                   (or real-ast ast)) ; real-ast comes from import-macros
  ;; don't require modname to be string literal; it just needs to compile to one
  (let [filename (or (. ast 2 :filename) ast.filename)
        modname-code (compiler.compile (. ast 2))
        modname ((load-code modname-code nil filename) utils.root.options.module-name
        modname-chunk (load-code (compiler.compile (. ast 2)) nil filename)
        modname (modname-chunk utils.root.options.module-name filename)]
    (compiler.assert (= (type modname) :string)
                     "module name must compile to string" (or real-ast ast))
    (when (not (. macro-loaded modname))
      (let [env (make-compiler-env ast scope parent)]
        (tset macro-loaded modname (compiler-env-domodule modname env ast))))
      (let [env (make-compiler-env ast scope parent)
            (loader filename) (search-macro-module modname 1)]
        (compiler.assert loader (.. modname " module not found.") ast)
        (tset macro-loaded modname (loader modname filename))))
    (add-macros (. macro-loaded modname) ast scope parent)))

(doc-special :require-macros [:macro-module-name]
@@ -1209,6 +1210,7 @@ Lua output. The module must be a string literal and resolvable at compile time."
 : current-global-names
 : load-code
 : macro-loaded
 : macro-searchers
 : make-compiler-env
 : search-module
 : make-searcher