I've been tinkering with my libraries for the last week or so, porting
them to the single file approach, where both macros and functions are
exported from the same file with a small eval-compiler and macro-loaded
hack.  For what it's worth it works fairly well, allowing me to
distribute my libraries easier than before.  And once again it got me
thinking how we can make macros more like macros in other lisps.

We've discussed in the past the ability to export macros from the same
file with some new special, and while I think this approach might work
there's one other elephant in the room - macros using other macros *and*
functions from the same module.  Consider:

``` fennel
(fn reverse [t]
  (fcollect [i (length t) 1]
    (. t i)))

(macro some-macro [x]
  ;; does some stuff at compile-time
  `(let [reversed# ,(reverse x)] ; needs to reverse at compile-time
     ;; does some stuff at runtime
     (reverse result#) ; needs to reverse at runtime
  ))

(fn foo [x]
  (some-macro x) ;; module uses the macro
  )

(export-macros
  { : some-macro})
{: reverse}
```

In this example, I want the macro to reuse some functions both at
compile-time and at run-time.

The first part should be relatively easy to implement - when compiling
the module to Lua we need to give macros the ability to reuse what's in
the module, so it should be similar to what's already happening when we
load a macro module.

The second part is trickier because the compiled result of the macro
right now looks like this:

```
repl>> (require-macros :our-module)
repl>> (macrodebug (some-macro (1 2 3 +)))
(let [reversed_1_auto (+ 3 2 1)]
  ;; does some stuff
  (reverse result_1_auto))
```

This is problematic because the call to reverse will reference a free
variable or something else entirely.

So I've been thinking about how other lisps deal with it, mainly looking
at Clojure, and its approach of fully qualifying symbols when expanding
backquoted lists.  They call it syntax quote[1] and I think we can do
the same because after we've loaded the module we actually can fully
qualify any public function of symbol in it.

We can do pretty much the same, and it will actually work similarly. For
reference, here's what Clojure does:

```clojure
user> (ns module)
nil
module> (defn public-foo [] :ok)
#'module/public-foo
module> (defn- private-bar [] :err)
#'module/private-bar
module> (defmacro foo [] `(public-foo))
#'module/foo
module> (defmacro bar [] `(private-bar))
#'module/bar
module> (in-ns 'user)
#namespace[user]
user> (require 'module)
nil
user> (macroexpand-1 '(module/foo))
(module/public-foo)
user> (macroexpand-1 '(module/bar))
(module/private-bar)
user> (module/foo)
:ok
user> (module/bar)
Syntax error (IllegalStateException) compiling module/private-bar at (*cider-repl localhost:45647(clj)*:46:3).
var: #'module/private-bar is not public
``

So despite the fact that we've just written `(public-foo) it expanded to
(module/public-foo).  And you can see that when the macro tries to call
a private function, we get an error.

Now, let's say we do the same in fennel:

```fennel
(fn public-foo [] :ok)
(fn private-bar [] :err)
(macro foo [] `(public-foo))
(macro bar [] `(private-bar))
(export-macros {: foo : bar})
{: public-foo}
```

Now if we repeat the same steps in the REPL, we should get this:

```
repl> (local module (require :module)) ; this actually sets up macro-loaded
nil
repl> (require-macros :module) ; this pulls from macro-loaded
nil
repl> (macrodebug (foo))
(package.loaded.module.public-foo)
repl> (macrodebug (bar))
(package.loaded.module.private-bar)
repl> (foo)
:ok
repl> (bar)
ERROR: attempt to call a nil value (field 'private-bar')
```

There are some things to consider:

First, we need to carefully manage macro-loaded.  It should be set up
properly either by doing (require :module) or by using
require-macros/import-macros.

Second, using macros with private definitions in the module itself
should work.  With my approach described above, if we were to try using
the bar macro in the file itself it would compile to
(package.loaded.module.private-bar) and would fail.  So I think when
using macros in the same module as they're defined there's no need to do
syntax-quote, and we can leave current behavior.

Finally, this means that macros will do repeated look-ups into the
package.loaded table.  This is also the way it is done currently, as
macros that need to reuse public functions tend to require the module in
the macro's generated body, but it is amortized as the lookup can be
done once per expansion, and identifiers can be used multiple times:

```
(macro x []
  `(let [{: a : b : c} (require :lib)]
     (a) (a)
     (b) (b)
     (c) (c)))
;; vs
(macro x []
  `(do
     (package.loaded.lib.a) (package.loaded.lib.a)
     (package.loaded.lib.b) (package.loaded.lib.b)
     (package.loaded.lib.c) (package.loaded.lib.c)))
```

So maybe some advanced code walking will be required.  I'm not sure what
is the cost of such namespace qualification in Clojure, but I assume
it's negligible due to JVM's JIT.  The same can be said of Luajit, but
it's not our only runtime.

Let me know what you think!

[1] https://clojure.org/reference/reader#syntax-quote

--
Andrey Listopadov

Andrey Listopadov <andreyorst@gmail.com> writes:

> I've been tinkering with my libraries for the last week or so, porting
> them to the single file approach, where both macros and functions are
> exported from the same file with a small eval-compiler and macro-loaded
> hack.  For what it's worth it works fairly well, allowing me to
> distribute my libraries easier than before.  And once again it got me
> thinking how we can make macros more like macros in other lisps.

Thanks for posting your thoughts on this. I agree that the current
status quo is not ideal, and we should work towards a solution which
makes things smoother for macro authors while preserving the
transparency of the existing system.

> The first part should be relatively easy to implement - when compiling
> the module to Lua we need to give macros the ability to reuse what's in
> the module, so it should be similar to what's already happening when we
> load a macro module.

Sorry, I don't think this is easy! This sounds really difficult, because
"what's in the module" will frequently depend on modules which will not load
in the compiler sandbox.

> So I've been thinking about how other lisps deal with it, mainly looking
> at Clojure, and its approach of fully qualifying symbols when expanding
> backquoted lists.  They call it syntax quote[1] and I think we can do
> the same because after we've loaded the module we actually can fully
> qualify any public function of symbol in it.
>
> We can do pretty much the same, and it will actually work similarly.

The big difference is that Clojure does this using vars, which are a
first-class construct of the language. When you say `def` in Clojure,
you're creating a storage location with a given name and value. I'd
go so far as to say that vars *are* the reason Clojure can do this.

The closest thing we have in Fennel (a storage location for functions
and other values) is modules, but it's very important to remember that
modules and locals basically only interact in two places: `require` and
exporting at the bottom of a module. (I know that you already know this,
Andrey; I'm just spelling it out in more detail for others on the list.)
When you define a `fn` in Fennel, it only affects local scope. It's not
until that local is placed in the export table that it interacts with
the module system. At the point the macro is defined, that hasn't
happened yet.

> ```fennel
> (fn public-foo [] :ok)
> (fn private-bar [] :err)
> (macro foo [] `(public-foo))
> (macro bar [] `(private-bar))
> (export-macros {: foo : bar})
> {: public-foo}
> ```

The problem with this is that it implies that the `foo` macro is pulling
`public-foo` out of its local scope, which is misleading. This kind of
connection is fragile and easy to confuse; for instance:

```fennel
(fn public-foo [] :ok)
(macro foo [] `(public-foo))
(fn public-foo [] :not-ok!!!)

(fn public-bar [] :ok)
(macro bar [] `(public-bar))

(export-macros {: foo : bar})
{: public-foo :bar public-bar}
```

This demonstrates that there are two different failure modes; in both
cases the local scope makes the macro look like it should expand to a
function which returns :ok, but in the first case the local scope
changes before we get to the export, and in the second the export
uses a different name from the local.

Clojure doesn't have this problem because the macro can make a direct
reference to the var as vars are effectively just globals.

Personally I think it's probably not a great idea to try to solve the
problem of export-macros at the same time as changing quoting. Let's
address these as two separate but related problems.

Previously I was a bit reluctant to have an export-macros form because I
really like the way that normal exports are all just listed at the
bottom of the file, but I think having an explicit export-macros form is
probably justified given the alternatives at this point. We should
explore in a little more detail what that means.

As for the issue of quoting, I'm not necessarily against the idea of
expanding out to a fully-qualified package.loaded reference. I just
think it needs to be clear at a glance that what's going on is different
from a local reference, and we can't change the meaning of existing
quote calls. We would need to come up with some kind of new notation to
indicate this.

-Phil

>Sorry, I don't think this is easy! This sounds really difficult, because
>"what's in the module" will frequently depend on modules which will not load
>in the compiler sandbox.

Honestly, I never fully understood why we do sandboxing of the compiler
in the first place.  It, however, limits interesting things that can be done with
macros, like conditional based on environment variable for example.

In fenneldoc I do the sandboxing mainly so the scripts to minimize effects, but
a lot of libraries may need to escape sandbox to test the documentation, so I provide
a setting for that. Though I think I'll drop the sandboxing in future release entirely.

>> ```fennel
>> (fn public-foo [] :ok)
>> (fn private-bar [] :err)
>> (macro foo [] `(public-foo))
>> (macro bar [] `(private-bar))
>> (export-macros {: foo : bar})
>> {: public-foo}
>> ```
>
>The problem with this is that it implies that the `foo` macro is pulling
>`public-foo` out of its local scope, which is misleading. This kind of
>connection is fragile and easy to confuse; for instance:
>
>```fennel
>(fn public-foo [] :ok)
>(macro foo [] `(public-foo))
>(fn public-foo [] :not-ok!!!)
>
>(fn public-bar [] :ok)
>(macro bar [] `(public-bar))
>
>(export-macros {: foo : bar})
>{: public-foo :bar public-bar}
>```
>
>This demonstrates that there are two different failure modes; in both
>cases the local scope makes the macro look like it should expand to a
>function which returns :ok, but in the first case the local scope
>changes before we get to the export, and in the second the export
>uses a different name from the local.
>
>Clojure doesn't have this problem because the macro can make a direct
>reference to the var as vars are effectively just globals.

Sorry, I don't get what's the problem here? I'd expect the foo macro
to return :not-ok!!!, same as Clojure would do.

>Personally I think it's probably not a great idea to try to solve the
>problem of export-macros at the same time as changing quoting. Let's
>address these as two separate but related problems.

I agree strongly. Just felt like sharing this to further explore the idea.

>Previously I was a bit reluctant to have an export-macros form because I
>really like the way that normal exports are all just listed at the
>bottom of the file, but I think having an explicit export-macros form is
>probably justified given the alternatives at this point. We should
>explore in a little more detail what that means.
>
>As for the issue of quoting, I'm not necessarily against the idea of
>expanding out to a fully-qualified package.loaded reference. I just
>think it needs to be clear at a glance that what's going on is different
>from a local reference, and we can't change the meaning of existing
>quote calls. We would need to come up with some kind of new notation to
>indicate this.

How about #' ?


-- 
Andrey Listopadov

Andrey <andreyorst@gmail.com> writes:

>> Sorry, I don't think this is easy! This sounds really difficult, because
>> "what's in the module" will frequently depend on modules which will not load
>> in the compiler sandbox.
>
> Honestly, I never fully understood why we do sandboxing of the compiler
> in the first place.  It, however, limits interesting things that can be done with
> macros, like conditional based on environment variable for example.

Well, have you taken a look at fennel-ls? If you compare the quality of
the static analysis from fennel-ls vs the Clojure language server, the
Fennel one is dramatically better despite being much less code. This is
only possible because it can do static analysis on the expanded code,
whereas the Clojure just gives up at the first sight of a 3rd-party macro.

> In fenneldoc I do the sandboxing mainly so the scripts to minimize
> effects, but a lot of libraries may need to escape sandbox to test the
> documentation, so I provide a setting for that. Though I think I'll
> drop the sandboxing in future release entirely.

Yes, the fact that the sandbox is all-or-nothing is definitely not
ideal. I've posted a sketch of some ideas to improve the situation on
the wiki: https://wiki.fennel-lang.org/CompilerSandboxGranularity

The main problem is that this configuration is complex enough that
specifying it with command-line arguments (currently the only
configuration the compiler accepts) is difficult and tedious. We may
need to start thinking about a configuration file format. Would love to
hear peoples' thoughts on this.

>> The problem with this is that it implies that the `foo` macro is pulling
>> `public-foo` out of its local scope, which is misleading. This kind of
>> connection is fragile and easy to confuse; for instance:
>>
>> ```fennel
>> (fn public-foo [] :ok)
>> (macro foo [] `(public-foo))
>> (fn public-foo [] :not-ok!!!)
>>
>> (fn public-bar [] :ok)
>> (macro bar [] `(public-bar))
>>
>> (export-macros {: foo : bar})
>> {: public-foo :bar public-bar}
>
> Sorry, I don't get what's the problem here? I'd expect the foo macro
> to return :not-ok!!!, same as Clojure would do.

That's only true because Clojure's `defn` is a destructive operation
which overwrites the current value of the var, while Fennel`s `fn` (when
not using a table) is non-destructive; all it does is add something to
the current lexical scope. The `foo` macro above looks like a closure,
but it's not.

>>As for the issue of quoting, I'm not necessarily against the idea of
>>expanding out to a fully-qualified package.loaded reference. I just
>>think it needs to be clear at a glance that what's going on is different
>>from a local reference, and we can't change the meaning of existing
>>quote calls. We would need to come up with some kind of new notation to
>>indicate this.
>
> How about #' ?

Hmm... We have single-quote as an undocumented alias for backtick; I
know that there are some codebases which use this, but they don't use it
in combination with #. Maybe if we can get the parser's prefix
implementation to have a bit of lookahead without too much hassle we
could use it? The other option of course is something involving @ which
we have reserved, but if we use it now then it might restrict what we
can do in the future.

-Phil

>Well, have you taken a look at fennel-ls? If you compare the quality of
>the static analysis from fennel-ls vs the Clojure language server, the
>Fennel one is dramatically better despite being much less code. This is
>only possible because it can do static analysis on the expanded code,
>whereas the Clojure just gives up at the first sight of a 3rd-party macro.

I wonder if this is because Fennel compiler is much easier to embed and use?
I haven't seen any public statements that clojure-lsp doesn't expand macros
due to security concerns. I suppose, Clojure macros are way harder to expand
if done manually, and it was too hard or cumbersome to rmbed the whole 
compiler for this sole job. Anyhow, I think clojure-lsp solution to this is fine, and
fennel-ls solution is also fine.

>> In fenneldoc I do the sandboxing mainly so the scripts to minimize
>> effects, but a lot of libraries may need to escape sandbox to test the
>> documentation, so I provide a setting for that. Though I think I'll
>> drop the sandboxing in future release entirely.
>
>Yes, the fact that the sandbox is all-or-nothing is definitely not
>ideal. I've posted a sketch of some ideas to improve the situation on
>the wiki: https://wiki.fennel-lang.org/CompilerSandboxGranularity

Interesting ideas, need to think about it more before I can comment on this.

>>> The problem with this is that it implies that the `foo` macro is pulling
>>> `public-foo` out of its local scope, which is misleading. This kind of
>>> connection is fragile and easy to confuse; for instance:
>>>
>>> ```fennel
>>> (fn public-foo [] :ok)
>>> (macro foo [] `(public-foo))
>>> (fn public-foo [] :not-ok!!!)
>>>
>>> (fn public-bar [] :ok)
>>> (macro bar [] `(public-bar))
>>>
>>> (export-macros {: foo : bar})
>>> {: public-foo :bar public-bar}
>>
>> Sorry, I don't get what's the problem here? I'd expect the foo macro
>> to return :not-ok!!!, same as Clojure would do.
>
>That's only true because Clojure's `defn` is a destructive operation
>which overwrites the current value of the var, while Fennel`s `fn` (when
>not using a table) is non-destructive; all it does is add something to
>the current lexical scope. The `foo` macro above looks like a closure,
>but it's not.

Sorry, but I fail to see how anything in a quoted list can look like a closure.
That's the point of quoting - to prevent any evaluation, thus no closures are
created at all. 

Same goes for clojure as well. Vars are introduced much later at the
evaluation state, macroexpanding doesn't see the var value change at all. It
just expands to a name in the parentheses, which is then resolved to its
current state during evaluation.

And then it works as you'd expect it to:

(fn foo [] (print :a))
(macro bar [] `(foo))
(bar) ;=> prints "a"
(fn foo [] (print :b))
(bar) ;=> prints "b"

And it's exactly how it works now, so nothing changes here.

>>>As for the issue of quoting, I'm not necessarily against the idea of
>>>expanding out to a fully-qualified package.loaded reference. I just
>>>think it needs to be clear at a glance that what's going on is different
>>>from a local reference, and we can't change the meaning of existing
>>>quote calls. We would need to come up with some kind of new notation to
>>>indicate this.
>>
>> How about #' ?
>
>Hmm... We have single-quote as an undocumented alias for backtick; I
>know that there are some codebases which use this, but they don't use it
>in combination with #. Maybe if we can get the parser's prefix
>implementation to have a bit of lookahead without too much hassle we
>could use it? The other option of course is something involving @ which
>we have reserved, but if we use it now then it might restrict what we
>can do in the future.

I still hope that unquote splicing will come, so I'd like to avoid confusion
when reading @' and ,@ (if it will use this syntax, that is), hence why I thought
about sharp quote instead, as it is something that already exists in lisps.

On a side note:
I have started working on a library that will implement custom quote and
unquote functions that, for now, will act on tables instead of lists. I want
to try and implement macros in terms of this and eval, and see if it will work
more robust in terms of being able to export macros, and I'll also experiment
with syntax quoting. But no promises for now, it's a hard thing to make properly.


-- 
Andrey Listopadov