Add plugin system, take 2. v1 PROPOSED

Phil Hagelberg: 1
 Add plugin system, take 2.

 8 files changed, 101 insertions(+), 5 deletions(-)
Export patchset (mbox)
How do I use this?

Copy & paste the following snippet into your terminal to import this patchset into git:

curl -s https://lists.sr.ht/~technomancy/fennel/patches/12147/mbox | git am -3
Learn more about email & git
View this thread in the archives

[PATCH] Add plugin system, take 2. Export this patch

This is roughly the same as the previous take on the plugin system,
which was based on the pre-self-hosted compiler.

Each plugin is a module which contains functions for various extension
points within the compiler. Each field in the module is named after an
extension point and should be a function that takes AST arguments and
a scope table. Current supported extension points are:

* symbol-to-expression
* call
* destructure
* do
* fn

These were chosen because they're the minimum needed to implement the
linting example, not by looking thru the whole compiler to find an
exhaustive list of good extension points. So we are likely to be
missing some good candidates for extension; we should be open to
adding new ones.

The included linter.fnl file shows an example of how the system works.

However, the problem here is that we are exposing compiler internals,
and that means that when we change the internals, we are likely to
break plugins when we refactor the compiler. I don't really know what
to do about that, and it seems inevitable. So we should probably just
try to set expectations that plugins will tend to be only compatible
with a limited set of compiler versions.

One problem is that the plugins need to be loaded in compiler scope,
which means we need a way to indicate in fennel.dofile that we
want to load the file in question in the compiler scope. In this patch
you can pass :env :_COMPILER as a special flag which causes the :env
to get replaced by a new compiler-scope environment. I don't love
this; I think we could come up with a better way to expose it.

We still need more documentation for this in api.md before the release.
 changelog.md            |  1 +
 reference.md            |  5 +++-
 src/fennel.fnl          |  6 ++++-
 src/fennel/compiler.fnl |  3 +++
 src/fennel/specials.fnl |  3 +++
 src/fennel/utils.fnl    |  8 +++++-
 src/launcher.fnl        |  8 ++++--
 src/linter.fnl          | 72 +++++++++++++++++++++++++++++++++++++++++++++++++
 8 files changed, 101 insertions(+), 5 deletions(-)
 create mode 100644 src/linter.fnl

diff --git a/changelog.md b/changelog.md
index 271c03f..c3fd71a 100644
--- a/changelog.md
+++ b/changelog.md
@@ -2,6 +2,7 @@

## 0.5.1 / ???

* Add plugin system.
* Fix a bug where repls would fail when provided with an overridden env.
* Expose `list?` and `sym?` in compiler API.
* Fix a bug where method calls would early-evaluate their receiver.
diff --git a/reference.md b/reference.md
index 1fd5e1e..6bf4dcc 100644
--- a/reference.md
+++ b/reference.md
@@ -993,7 +993,7 @@ Example:

This prints all the functions available in compiler scope.

### Compiler API
### Compiler Environment

Inside `eval-compiler`, `macros`, or `macro` blocks, as well as
`import-macros` modules, these functions are visible to your code.
@@ -1016,6 +1016,9 @@ and a metatable that the compiler uses to distinguish them. You can use
* `varg?` - is this a `...` symbol which indicates var args?
* `multi-sym?` - a multi-sym is a dotted symbol which refers to a table's field

* `assert-compile` - works like `assert` but takes a list/symbol as its third
  argument in order to provide pinpointed error messages.

These functions can be used from within macros only, not from any
`eval-compiler` call:

diff --git a/src/fennel.fnl b/src/fennel.fnl
index 877c256..7b34ac1 100644
--- a/src/fennel.fnl
+++ b/src/fennel.fnl
@@ -35,7 +35,11 @@
        _ (when (and (= opts.allowedGlobals nil)
                     (not (getmetatable opts.env)))
            (set opts.allowedGlobals (specials.current-global-names opts.env)))
        env (and opts.env (specials.wrap-env opts.env))
        ;; This is ... not great. Should we expose make-compiler-env in the API?
        env (if (= opts.env :_COMPILER)
                (specials.wrap-env (specials.make-compiler-env
                                    nil compiler.scopes.compiler {}))
                (and opts.env (specials.wrap-env opts.env)))
        lua-source (compiler.compile-string str opts)
        loader (specials.load-code lua-source env
                                  (if opts.filename
diff --git a/src/fennel/compiler.fnl b/src/fennel/compiler.fnl
index 872797a..b79f80c 100644
--- a/src/fennel/compiler.fnl
+++ b/src/fennel/compiler.fnl
@@ -187,6 +187,7 @@ rather than generating new one."
(fn symbol-to-expression [symbol scope reference?]
  "Convert symbol to Lua code. Will only work for local symbols
if they have already been declared via declare-local"
  (utils.hook :symbol-to-expression symbol scope reference?)
  (let [name (. symbol 1)
        multi-sym-parts (utils.multi-sym? name)
        name (or (hashfn-arg-name name multi-sym-parts scope) name)]
@@ -412,6 +413,7 @@ if opts contains the nval option."

(fn compile-call [ast scope parent opts compile1]
  (utils.hook :call ast scope)
  (let [len (# ast)
        first (. ast 1)
        multi-sym-parts (utils.multi-sym? first)
@@ -678,6 +680,7 @@ which we have to do if we don't know."
        {:returned true}))

    (let [ret (destructure1 to nil ast true)]
      (utils.hook :destructure from to scope)
      (apply-manglings scope new-manglings ast)

diff --git a/src/fennel/specials.fnl b/src/fennel/specials.fnl
index ee36099..f77dba9 100644
--- a/src/fennel/specials.fnl
+++ b/src/fennel/specials.fnl
@@ -120,6 +120,7 @@ By default, start is 2."
              fargs (if scope.vararg "..." "")]
          (compiler.emit parent (string.format "local function %s(%s)"
                                               fname fargs) ast)
          (utils.hook :do ast sub-scope)
          (compile-body nil true
                        (utils.expr (.. fname "(" fargs ")") :statement))))))

@@ -227,6 +228,7 @@ the number of expected arguments."
      (compiler.emit parent f-chunk ast)
      (compiler.emit parent "end" ast)
      (set-fn-metadata arg-list docstring parent fn-name))
    (utils.hook :fn ast f-scope)
    (utils.expr fn-name "sym")))

(doc-special "fn" ["name?" "args" "docstring?" "..."]
@@ -826,6 +828,7 @@ Method name doesn't have to be known at compile-time; if it is, use

                 :fennel utils.fennel-module
                 :unpack unpack
                 :assert-compile compiler.assert

                 ;; AST functions
                 :list utils.list
diff --git a/src/fennel/utils.fnl b/src/fennel/utils.fnl
index 4c9a176..75802e2 100644
--- a/src/fennel/utils.fnl
+++ b/src/fennel/utils.fnl
@@ -235,6 +235,12 @@ has options calls down into compile."
    (set (root.chunk root.scope root.options root.reset)
         (values chunk scope options reset))))

(fn hook [event ...]
  (when (and root.options root.options.plugins)
    (each [_ plugin (ipairs root.options.plugins)]
      (match (. plugin event)
        f (f ...)))))

{;; general table functions
 : allpairs : stablepairs : copy : kvmap : map : walk-tree : member?

@@ -243,6 +249,6 @@ has options calls down into compile."
 : expr? : list? : multi-sym? : sequence? : sym? : table? : varg? : quoted?

 ;; other
 : valid-lua-identifier? : lua-keywords
 : valid-lua-identifier? : lua-keywords : hook
 : propagate-options : root : debug-on?
 :path (table.concat ["./?.fnl" "./?/init.fnl" (getenv "FENNEL_PATH")] ";")}
diff --git a/src/launcher.fnl b/src/launcher.fnl
index 80b2810..0d6742b 100644
--- a/src/launcher.fnl
+++ b/src/launcher.fnl
@@ -40,7 +40,7 @@ Run fennel, a lisp programming language for the Lua runtime.

  If ~/.fennelrc exists, loads it before launching a repl.")

(local options [])
(local options {:plugins []})

(fn dosafely [f ...]
  (let [args [...]
@@ -106,7 +106,11 @@ Run fennel, a lisp programming language for the Lua runtime.
    "--metadata" (do (set options.useMetadata true)
                     (table.remove arg i))
    "--no-metadata" (do (set options.useMetadata false)
                        (table.remove arg i))))
                        (table.remove arg i))
    "--plugin" (let [plugin (fennel.dofile (table.remove arg (+ i 1))
                                           {:env :_COMPILER})]
                 (table.insert options.plugins 1 plugin)
                 (table.remove arg i))))

(when (not options.no_searcher)
  (let [opts []]
diff --git a/src/linter.fnl b/src/linter.fnl
new file mode 100644
index 0000000..6712f82
--- /dev/null
+++ b/src/linter.fnl
@@ -0,0 +1,72 @@
;; An example of some possible linters using Fennel's --plugin option.

;; The linters here can only function on static module use. For instance, this
;; code can be checked because they use static field access on a local directly
;; bound to a require call:

;; (local m (require :mymodule))
;; (print m.field) ; fails if mymodule lacks a :field field
;; (print (m.function 1 2 3)) ; fails unless mymodule.function takes 3 args

;; However, these cannot:

;; (local m (do (require :mymodule)) ; m is not directly bound
;; (print (. m field)) ; not a static field reference
;; (let [f m.function]
;;   (print (f 1 2 3)) ; intermediate local, not a static field call on m

;; Still, pretty neat, huh?

(fn save-require-meta [from to scope]
  "When destructuring, save module name if local is bound to a `require' call.
Doesn't do any linting on its own; just saves the data for other linters."
  (when (and (sym? to) (not (multi-sym? to)) (list? from)
             (sym? (. from 1)) (= :require (tostring (. from 1)))
             (= :string (type (. from 2))))
    (let [meta (. scope.symmeta (tostring to))]
      (set meta.required (tostring (. from 2))))))

(fn check-module-fields [symbol scope]
  "When referring to a field in a local that's a module, make sure it exists."
  (let [[module-local field] (or (multi-sym? symbol) [])
        module-name (and module-local (. scope.symmeta
                                         (tostring module-local) :required))
        module (and module-name (require module-name))]
    (assert-compile (or (= module nil) (not= (. module field) nil))
                    (string.format "Missing field %s in module %s"
                                   field module-name) symbol)))

(fn arity-check? [module] (-?> module getmetatable (. :arity-check?)))

(fn arity-check-call [[f & args] scope]
  "Perform static arity checks on static function calls in a module."
  (let [arity (# args)
        last-arg (. args arity)
        [f-local field] (or (multi-sym? f) [])
        module-name (and f-local (. scope.symmeta (tostring f-local) :required))
        module (and module-name (require module-name))]
    (when (and (arity-check? module) debug debug.getinfo
               (not (varg? last-arg)) (not (list? last-arg)))
      (assert-compile (= (type (. module field)) :function)
                      (string.format "Missing function %s in module %s"
                                     field module-name) f)
      (match (debug.getinfo (. module field))
        {: nparams :what "Lua" :isvararg true}
        (assert-compile (<= nparams (# args))
                        (: "Called %s.%s with %s arguments, expected %s+"
                           :format f-local field arity nparams) f)
        {: nparams :what "Lua" :isvararg false}
        (assert-compile (= nparams (# args))
                        (: "Called %s.%s with %s arguments, expected %s"
                           :format f-local field arity nparams) f)))))

(fn check-unused [ast scope]
  (each [symname (pairs scope.symmeta)]
    (assert-compile (or (. scope.symmeta symname :used) (symname:find "^_"))
                    (string.format "unused local %s" symname) ast)))

{:destructure save-require-meta
 :symbol-to-expression check-module-fields
 :call arity-check-call
 :fn check-unused
 :do check-unused}