A few days ago I had a magnificent premonition.  I saw how macros and
functions coexist in the same module, and I can do both (local fns
(require :mod)) and (import-macros macs :mod) and everything worked
fine.  Then I woke up...

...and did it.

The trick involves the eval-compiler special and the fact that
macro-loaded exists.

Here's an example library:

(eval-compiler
  (fn when-not [test ...]
    `(when (not ,test)
       ,...))
  (tset macro-loaded :some-lib {: when-not}))

(import-macros {: when-not} :some-lib)

(fn countdown [i]
  (when-not (= i 0)
    (print i)
    (countdown (- i 1))))

{: countdown}

Now, in the REPL I can do this:

repl>> (local lib (require :some-lib))
nil
repl>> (lib.countdown 3)
3
2
1
nil
repl>> (import-macros {: when-not} :some-lib)
nil
repl>> (when-not false :works)
"works"

This doesn't require any changes in the compiler, and seems portable.

The only problem is that if the some-lib.fnl is renamed to
anything-else.fnl Macros will still be using the some-lib namespace.  We
could migrate it by caputing the vararg on the top level of the library,
but it's impossible to access that in the eval-compiler scope,
unfortunately:

(local lib-name ...)
(eval-compiler
  (fn when-not [test ...]
     `(when (not ,test)
        ,...))
  (tset macro-loaded lib-name {: when-not}))
;; ----8<----
Compile error: unknown identifier: lib-name
  (tset macro-loaded lib-name {: when-not}))

And the vararg inside the eval-compiler is nil.  If it was possible to
pass the top-level ... somehow to eval-compiler, it would work
seamlessly.  On the other hand I can always assert for the library name:

(assert (= ... :some-lib) "must require as :some-lib")

What I like about this is that there's no need to change anything in the
compiler, and it still gives us the ability to both use macros in the
same lib and provide them to users in a convenient way as a single file.

Let me know what you think!  Maybe we can hack in another special that
will just wrap this around and do it like that, without any significant
changes to the compiler.

--
Andrey Listopadov

Now, I must add, that when the kibrary wan'ts to reuse macros it exports
everything is pretty simple.  However, if macros need to expand into calls
to library functions, the setup gets a bit more complicated.

Macros still have to require the library in their body, and that's why
knowing library name is important.

(eval-compiler
  (fn log/error [...]
    `(when (> _G.logging-level 1)
       (let [{:log log#} (require :log4f)]
         (log# :error ,...))))
  (fn log/warn [...]
    `(when (> _G.logging-level 1)
       (let [{:log log#} (require :log4f)]
         (log# :error ,...))))
  (fn do1 [...] `(do (print :useless-do) ,...))
  (tset macro-loaded :log4f {: log/error : log/warn : do1}))

(import-macros {: do1} :log4f)

(fn log [level ...]
  (do1 (io.write level ":" ... "\n")))

{: log}

E.g. here, if the library wasn't required as log4f macros will not work,
because there's no package.loaded instance of the library at that point.
-- 
Andrey Listopadov

Phil, can you point me in the right direction here? I'm searching why can't 
eval-compiler have an access to the module name from the fennel's 
searcher, and I don't think I see what's going on here as a whole.

As far as I can see the opts table contains the module name, so we should
be able to reach it from eval-compiler.  Having it reachable solves the
same-dile macro problem for 75% I think, so I'm really eager at tackling it.
-- 
Andrey Listopadov

I've now realized that this won't work outside of the REPL, so the idea
is now scraped.

The reason this won't work is because requiring the module is what set's
macros to the macro-loaded, and when used inside the file import-macros
will be compiled before the require call and thus will fail to find the
module.

That's an unfortunate outcome, I guess we'll have to go back to the
export-macros thing proper.

--
Andrey Listopadov

Andrey <andreyorst@gmail.com> writes:

> Phil, can you point me in the right direction here? I'm searching why can't 
> eval-compiler have an access to the module name from the fennel's 
> searcher, and I don't think I see what's going on here as a whole.

Hey, sorry for the delay on this. I'm in the middle of the lisp game jam
and so probably won't be catching up with most stuff until it ends.

It turns out this was a very simple mistake in the eval-compiler special:

diff --git a/src/fennel/specials.fnl b/src/fennel/specials.fnl
index b044704..503ddd9 100644
--- a/src/fennel/specials.fnl
+++ b/src/fennel/specials.fnl
@@ -1363,8 +1363,8 @@ Lua output. The module must be a string literal and resolvable at compile time."
         opts (utils.copy utils.root.options)]
     (set opts.scope (compiler.make-scope compiler.scopes.compiler))
     (set opts.allowedGlobals (current-global-names env))
-    ((assert (load-code (compiler.compile ast opts) (wrap-env env))
-             opts.module-name ast.filename))))
+    ((assert (load-code (compiler.compile ast opts) (wrap-env env)))
+     opts.module-name ast.filename)))
 
 (fn SPECIALS.macros [ast scope parent]
   (compiler.assert (= (length ast) 2) "Expected one table argument" ast)

The module name was being used as the assert message rather than the
argument passed to the loaded chunk. Once we make that fix, then
eval-compiler can see the module name in the vararg. I've pushed it out!

-Phil

Phil Hagelberg <phil@hagelb.org> writes:

> The module name was being used as the assert message rather than the
> argument passed to the loaded chunk. Once we make that fix, then
> eval-compiler can see the module name in the vararg. I've pushed it out!

Also, this is a very clever solution to the problem; nice work!

I would like to add a way to do this which doesn't require
eval-compiler, since that form is something of a measure of last resort,
but until we figure out a better way, this is a neat trick.

-Phil

> Hey, sorry for the delay on this. I'm in the middle of the lisp game jam
> and so probably won't be catching up with most stuff until it ends.

I was traveling lately and totally forgot about the jam!
Curious to see what entries will be this year so good luck!

> It turns out this was a very simple mistake in the eval-compiler special:
>
> diff --git a/src/fennel/specials.fnl b/src/fennel/specials.fnl
> index b044704..503ddd9 100644
> --- a/src/fennel/specials.fnl
> +++ b/src/fennel/specials.fnl
> @@ -1363,8 +1363,8 @@ Lua output. The module must be a string literal and resolvable at compile time."
>          opts (utils.copy utils.root.options)]
>      (set opts.scope (compiler.make-scope compiler.scopes.compiler))
>      (set opts.allowedGlobals (current-global-names env))
> -    ((assert (load-code (compiler.compile ast opts) (wrap-env env))
> -             opts.module-name ast.filename))))
> +    ((assert (load-code (compiler.compile ast opts) (wrap-env env)))
> +     opts.module-name ast.filename)))
>
>  (fn SPECIALS.macros [ast scope parent]
>    (compiler.assert (= (length ast) 2) "Expected one table argument" ast)
>
> The module name was being used as the assert message rather than the
> argument passed to the loaded chunk. Once we make that fix, then
> eval-compiler can see the module name in the vararg. I've pushed it out!

Lol, I've been staring at this code for hours and didn't noticed that.
Thanks a lot!

--
Andrey Listopadov

> I've now realized that this won't work outside of the REPL, so the idea
> is now scraped.
>
> The reason this won't work is because requiring the module is what set's
> macros to the macro-loaded, and when used inside the file import-macros
> will be compiled before the require call and thus will fail to find the
> module.

Correction it does work with another hack!

It requires re-requiring the module inside the `import-macros` form.
I've set up a branch[1] of the new asyncrhonous library to test tis and
loading it like this seems to work fine:

(local async (require :src.async))
(import-macros {: go : go-loop} (doto :src.async require))

Though I'm unsure if it should be done this way in real codebases.

[1] https://gitlab.com/andreyorst/async.fnl/-/commit/1ddeaad3286c2779dae662c927be1edd21fbe434

--
Andrey Listopadov

After a bit more fiddling, I rewrote my testing library to be a
single-file module, and I'm happy to say that the trick works both for
regular modules, and for init.fnl modules, which means that it is more
general than the relative-require stuff I was going for originally to
have a similar setup.

I'm pretty happy with the solution, and I think we can safely add a form
that will export its last value in a same way how the eval-compiler
thing does.

Phil, if you're cool with that, I'll see if I can make it work.

--
Andrey Listopadov

Andrey Listopadov <andreyorst@gmail.com> writes:

> After a bit more fiddling, I rewrote my testing library to be a
> single-file module, and I'm happy to say that the trick works both for
> regular modules, and for init.fnl modules, which means that it is more
> general than the relative-require stuff I was going for originally to
> have a similar setup.

I've also ported few other libraries to this style: async.fnl[1],
lazy-seq[2] and finally cljlib[3].  All of these, can now be required to
get functions, and the same module can then be required to get macros:

(local clj (require :cljlib))
(import-macros {: defn} (doto :cljlib require))

With cljlib it was a bit trickier, as it embeds other libraries, that
embed macros in this style, so I written a post going into the details
of how all of this work:

https://andreyor.st/posts/2023-08-27-fennel-libraries-as-single-files/

I would appreciate if someone would give me feedback on the approach,
after considering all of the problems outlined in the post.

[1]: https://gitlab.com/andreyorst/async.fnl
[2]: https://gitlab.com/andreyorst/lazy-seq
[3]: https://gitlab.com/andreyorst/fennel-cljlib

--
Andrey Listopadov