The next Fennel version

Message ID
DKIM signature
Download raw message
Hello Fennelers!

I wanted to write up what I've been thinking about the next
release. There have been a few nice bug fixes on the main branch
recently, but there are two main big new features that will be the
highlights of 0.6.0:

The first is something we've been thinking about for a while: the plugin
API. We want a way to extend the compiler beyond just macros. With the
plugin API, you can define callbacks which will get hit at certain
points in the compilation cycle.

One example of how that could be used is for linting. This module shows
how a linter could check for certain problems like unused locals or
attempting to load fields on a module which don't exist:


Another example could be translating code from another foreign language:


The main concern with the plugin API is that it exposes details about
the compiler internals, and so it will cause plugins to break when those
internals change. I'm not really sure how to address this other than to
start out by saying that this API is very experimental, and if you write
a plugin, you should not expect it to work with future versions; it will
need to be updated as the compiler is updated.

You can check out the "plugin2" branch to see the progress on this. It
still needs documentation but other than that I'm pretty happy with
where it is. I might also add a warning system so plugins can declare
themselves compatible with specific Fennel versions given that we expect
a lot of incompatibility.

The other big change is that we want to move towards sandboxing the
compiler by default. Making it so the code that runs in compiler scope
(macros and calls to eval-compiler) does not have access to the full
environment makes for much safer defaults. For instance, right now in
the pengbot IRC bot, we had to disable macros because there was no safe
way to run user-provided code.

(local safe-compiler-env
       (setmetatable {: table : math : string : pairs : ipairs : assert : error
                      : select : tostring : tonumber : pcall : xpcall : next
                      : print : type :bit _G.bit : setmetatable : getmetatable
                      :require safe-metadata-require}
                     {:__index compiler-env-warn}))

What we have right now is a restricted sandbox which allows access to
"pure" globals listed above but not things like io and os which can
affect the outside system. The biggest problem here is require... we
have to allow it because the docstring system uses it, but it's an easy
way to escape the sandbox. Right now we include a shim require which
lets thru the Fennel module's metadata table for docstrings but warns on
every other table.

It's difficult for me to guess how disruptive of a change this will be
since I don't have any of my own codebases that define any macros. If
you can try this for yourself I'd be interested in hearing how it goes
with the warnings. Of course the sandboxing can be disabled easily
enough if you intentionally write macros that need access to launch
shell scripts or do anything else wild, but it seems prudent not to
allow that as the default.

Would love to hear feedback about anything regarding the next version.

Export thread (mbox)