Hi everyone. First of all, thank you to Andrey for recently completing 
the new Proto REPL. It has really improved my Fennel workflow in Emacs.

I'm also wondering what the state of "hot reloading" support for Fennel 
is in general. I've peeked through the mailing list, taken a look at 
jeejah, and seen videos of people doing interesting things with Löve, 
but it seems like there's still some area for exploration here. 
Specifically:

1. Is there anything fundamental about the Lua runtime that prevents 
hot-reloading in the tradition of Sly/Slime or nREPL? (re: "patching 
satellites in prod")
2. Any there any existing tools or frameworks that already do this? (re: 
things in Lua; does Löve actually do this?)
3. Is there desire for a revival of jeejah or reimplementation of an 
nREPL for Fennel?

Thanks for any insight you can offer.

Colin

Colin Woodbury <colin_woodbury@caddi.jp> writes:

> Hi everyone. First of all, thank you to Andrey for recently completing
> the new Proto REPL. It has really improved my Fennel workflow in
> Emacs.

Glad to hear it, thanks!

> I'm also wondering what the state of "hot reloading" support for
> Fennel is in general. I've peeked through the mailing list, taken a
> look at jeejah, and seen videos of people doing interesting things
> with Löve, but it seems like there's still some area for exploration
> here. Specifically:
>
> 1. Is there anything fundamental about the Lua runtime that prevents
> hot-reloading in the tradition of Sly/Slime or nREPL? (re: "patching
> satellites in prod")

Yes, there's something fundamentally different in Fennel's REPL
implementation that makes the abitity to reload code a fair bit harder.

With nREPL you start the server, and it works in the background of your
application.  This is already hard, as Lua doesn't have the facilities
to start something in another thread.

A typical workflow in the case of nREPL/Sly, is that you can load
modules, and also you can re-define existing functions within those
modules and the program picks the new definition automatically without
requiring full module reload.  This is achieved by using vars in
Clojure, and slots (I believe) in CL.  Private definitions are simply
hidden from the user, though if you really want you can still access
them, and moreover, redefine them.

Fennel's REPL, however, doesn't know anything about modules.  When
you're doing (require :some-module), everything is basically baked
during the loading process in the package.loaded table.  Re-defining the
function in Fennel requires also redefining all of the functions that
use the old definition, because Lua uses closures to the locals of the
environment when the module was loaded.  And private definitions aren't
accessible outside of the module without the debug library trickery.

Hence, the only way to update the definition of anything that came from
the module is to remove its entry from the package.loaded, and
re-require it.  That's what the ,reload command in the REPL does.

> 2. Any there any existing tools or frameworks that already do this?
> (re: things in Lua; does Löve actually do this?)

As far as I know, only the ,reload command works reliably across
different systems.  E.g. if you check the minimal love2d fennel repo the
workflow there is based on reloading modules.

Alternatively, you /can/ send the entirety of your code to the REPL and
evaluate it there, but the REPL has only one module space available to
it, so functions and locals with the same name will clash and be
overridden by a more recent definition.  That's the workflow I use most
of the time when developing, but again, my projects usually consist of a
single module.

> 3. Is there desire for a revival of jeejah or reimplementation of an
> nREPL for Fennel?

JeeJah has the problem of Lua not having means of parallel execution.
It is possible to do a non-blocking REPL only if there's a way to read
the user input in a non-blocking way, which is hard, and not really
portable.  This can be achieved with sockets, however, it is still not
great, as the runtime has to do a continuous polling of the socket.
Additionally, sockets in Lua are not in the standard library, and
therefore JeeJah may not work in environments that don't provide a
socket library.

I believe Phil can give more details on why JeeJah is hard to make
properly and correct me if I stated something incorrectly :)

--
Andrey Listopadov

Andrey Listopadov <andreyorst@gmail.com> writes:

>> I'm also wondering what the state of "hot reloading" support for
>> Fennel is in general. I've peeked through the mailing list, taken a
>> look at jeejah, and seen videos of people doing interesting things
>> with Löve, but it seems like there's still some area for exploration
>> here. Specifically:
>>
>> 1. Is there anything fundamental about the Lua runtime that prevents
>> hot-reloading in the tradition of Sly/Slime or nREPL? (re: "patching
>> satellites in prod")

Andrey has written a lot of detail here which is great, but I'll provide
a simplified answer.

Reloading modules works great; it's 100% reliable, with a simple
implementation that's available everywhere. If you rely on ,reload for
your interactive development, everything will be nice and smooth.

What you can't do is "enter" a module the way you would in a repl for
Clojure or some other lisps. The top-level is its own chunk, and it
lives outside a module. You can take all the code for a module and dump
it into the top-level, but it's not going to *be* the module; it will be
its own distinct thing.

This can be frustrating if you try to use it the same way as
Clojure. Don't do that! Just ... require the module from the repl, and
reload the module when you make changes.

I've gone into more detail about how this works on my blog:

https://technomancy.us/189

>> 2. Any there any existing tools or frameworks that already do this?
>> (re: things in Lua; does Löve actually do this?)

Yes, the implementation of reloading modules is very simple, and there
have been plenty of independent reimplementations of it. The most
popular is the `hotswap` function from the widely-used lume library:

https://github.com/rxi/lume#lumehotswapmodname

>> 3. Is there desire for a revival of jeejah or reimplementation of an
>> nREPL for Fennel?
>
> I believe Phil can give more details on why JeeJah is hard to make
> properly and correct me if I stated something incorrectly :)

This is an interesting question because it can mean so many different
things, and depending on how you interpret the question the answer is
different. =)

1) Jeejah's actual repl implementation: absolutely not; it's very
   rudimentary and primitive. Fennel's built-in repl is better in
   every way.

2) Jeejah as a method of exposing a repl over a network: ehhhh, no, not
   really. Andrey has a networked repl in his async library if you want
   to embed in an application; if you just want a bare repl outside an
   event loop then it's simpler to use netcat.

3) Jeejah as a pluggable, extensible evaluation protocol: this is where
   it starts to get a little more interesting; the nREPL protocol allows
   you to define custom operations and handlers that clients can invoke;
   for instance, there is a completion command which sends a custom op
   rather than sending code to be evaluated remotely, which allows the
   completion logic to live in the Fennel side, where it can send a
   structured list of targets back to the server. This is very neat, but
   I think the proto-repl is probably a better fit for this, except for
   one factor which brings us to the next point.

4) Jeejah as an implementation of the standardized nREPL protocol:
   really the one thing it has going for it is the standardized
   nature. You can write one nREPL server and immediately get support
   for scores of editors for free, similarly to LSP. This is pretty
   compelling if you don't use Emacs or Vim! But it does mean it's not
   very compelling to me personally.

So I'm more or less done with Jeejah, but I still see it as a project
which potentially could have value for the wider community.
Unfortunately while the nREPL protocol's original design emphasized
being language-agnostic, the current maintainers don't really prioritize
this, which has led to most clients making clojure-specific assumptions
leading to incompatibility. Clients which work with Fennel (or any
non-Clojure language) are somewhat rare.

So while I believe this has a lot of potential, it's also an uphill
battle trying to work against the momentum of the mistakes that have
been made in the ecosystem.

Hope that makes sense!

-Phil

> Don't do that! Just ... require the module from the repl, and
> reload the module when you make changes.
>
> I've gone into more detail about how this works on my blog:
>
> https://technomancy.us/189

I would like to note that Fennel doesn't have something like Clojure's
defonce, so when reloading the module all of the state, stored in the
module is lost.

The state can be put into another module, which may be a bit more
complicated than one might like, but it works.

One case that makes this problematic is when the module introduces some
singleton, that is used for testing if the object is the same as the
singleton.  For example, the following module defines an Obj singleton,
and a function to create one, and test against:

(local Obj {})

(fn make-obj [foo bar]
  (setmetatable {: foo : bar} Obj))

(fn obj? [obj]
  (rawequal (getmetatable obj) Obj))

{: make-obj : obj?}


If in the REPL one does the following:

repl>> (local obj (require :obj))
{:make-obj #<fn> :obj? #<fn>}
repl>> (local some-obj (obj.make-obj :a :b))
nil

And then reload the obj module, the test would fail:

repl>> ,reload obj
ok
repl>> (obj.obj? some-obj)
false

This happens because some-obj still has a reference to the old instance
of the Obj table.  Reloading the module recreated that table, so it's a
different object in the VM.

So if you're doing some lengthy experiments in the REPL or iterative
development in general, beware that this may happen (happened to me just
the other day).

BTW, maybe defonce actually can be implemented? It should be as easy as:

(macro defonce [name val]
  `(local ,name
     (case _G.___replLocals___.state
       state# (match (?. state# ... ,(tostring name))
                {:val val#} val#
                ,(sym :_) (let [val# ,val]
                            (doto state#
                              (tset ... (or (. state# ...) {}))
                              (tset ... ,(tostring name) {:val val#}))
                            val#))
       nil ,val)))

(fn unintern [module name]
  (tset _G.___replLocals___.state module name nil))

If the ___replLocals___ has the state storage, we can track locals that
have been defonced on a per-module basis.  Reloading the module checks
if the module already has the local, and re-uses the value, much like
require caches the results in the package.loaded.  The macro doesn't
support destructuring though.

--
Andrey Listopadov

Thank you both for your detailed answers. For the time being, it seems 
like ,reload is the way forward for live testing. I've definitely been 
bitten by reloading functions one at a time, but having certain other 
transitive functions be stale.

 > Andrey has a networked repl in his async library if you want to embed 
in an application

Yes I was innocently wondering how feasible it is to experiment with 
live running programs. Can the proto repl be embedded? I suppose I'm not 
married to the idea of nREPL per se, more the idea of _some_ protocol 
for accessing a running program. Then again, as Andrey points out, we 
have issues of threading.

Colin

>Yes I was innocently wondering how feasible it is to experiment with live running programs. Can the proto repl be embedded? I suppose I'm not married to the idea of nREPL per se, more the idea of _some_ protocol for accessing a running program.

Yes, but there's no need to.

The proto-repl is designed to be injected into an ordinary fennel REPL, so all
that needs to be done is that the regular REPL has to be embedded into the
application, and then the client will send the upgrade code to it when
connected.

This may require defining a custom user input handler for io.read to work
if the REPL is running over the network, but the protocol supports READ OP
redefinition after the connection is established.

> Then again, as Andrey points out, we have issues of threading.

I'm using proto-repl with LOVE 2D, and it works nice, because the engine has
threads, and we can offload input reading to a separate thread. The REPL
itself is customized to be asynchronous, and proto-repl can still upgrade it
because of how the Fennel REPL is designed.

-- 
Andrey Listopadov

Andrey <andreyorst@gmail.com> writes:

> I'm using proto-repl with LOVE 2D, and it works nice, because the engine has
> threads, and we can offload input reading to a separate thread. The REPL
> itself is customized to be asynchronous, and proto-repl can still upgrade it
> because of how the Fennel REPL is designed.

Embedding the repl (whether proto- or not) is quite easy; the difficulty
is getting input to it, since there is no one-size-fits-all way to read
input. In the case of love2d it's relatively easy since you can do a
regular io.read on another thread, (tho it's still a little tricky to
set up channels to convey it to the main thread) but it's rare for lua
environments to support multiple threads like this.

Depending on what libraries you have available, you may be able to do a
non-blocking read from stdin, but often you don't necessarily want to
read from there; sometimes it makes more sense to read input from inside
the application itself. By running the repl in a coroutine and using
coroutine.yield as your read function, you can hook it into
application-specific input.

I wrote an article on the wiki that covers this; it gives an example of
how to do it for the Lite text editor. The first half covers some
unrelated topics, but the second half explains how to tie Lite's input
framework into the stock fennel.repl functionality:

https://wiki.fennel-lang.org/Lite

-Phil