I was messing around a bit with the repl over the weekend, plugging it
into DAP (the debugger adapter protocol similar to LSP) and I noticed
something frustrating. It's very easy to start a REPL that isn't
connected to standard IO (essentially a terminal) by overriding the
`readChunk`, `onValues`, and `onError` functions in the options table. But
unfortunately the `readChunk` function makes some assumptions about how
its input is coming in.

Being an API that was designed around line-based input, the `readChunk`
function returns a "chunk" of text at a time (unrelated to Lua's
definition of "chunk" of compiled code) gets passed to an iterator which
loops over the bytes until it runs out, then calls `readChunk` again for
another string of input.

This is a suitable API for a terminal, because input comes in one line
at a time, and no assumptions are made that each chunk will contain an
entire form, or that they won't contain multiple forms. However, when
you embed a REPL in another program, often the input does not come in
from an interface that gives you a line at a time. For instance, the
REPL in Emacs does not send anything to Fennel until it has a complete
expression. Similarly the DAP client will expect its input to be
well-formed before sending it to Fennel's REPL.

In a stdin-based setting, your `readChunk` function (whether custom or
default) will run when the user presses enter, but in this case enter is
being overloaded to mean two things: it's submitting what you've typed
so far, but it's *also* adding a newline to the form. This newline can
be the only thing that tells the parser that the form is complete; for
instance, if the user typed "123", then enter, then "456", the newline
is how the parser knows that there should be two separate numbers
instead of 123456 where there the "chunking" of the input just happened
to land in the middle.

The problem is that it's the responsibility of the custom `readChunk`
function to insert this newline. If you leave it out, you will end up
with 123456 where you wanted 123 and 456 as separate numbers. This is
really easy to overlook, because with most custom REPLs, submitting the
input and adding a newline are *different operations*.

I don't think this can be solved by changing the way the `readChunk` return
value is used; at least not without introducing backwards incompatibility.
In fact, the only backwards-compatible fix I can think of is to add a
new `opts.read-form` function, but all that would do is the exact same
thing as `opts.readChunk` except move the responsibility for adding the
newline from the caller to Fennel. I think it's hard to justify adding a
whole new option just for that. So maybe the only solution is to emphasize
the need to add the newline in the documentation?

I've pushed out an update to the docs explaining this, but I wanted to
write this up so I could find it later to make sure I remembered my
thought process. =)

It's noteworthy that the root cause of this problem is the conflation of
the two different meanings of the enter key. Just another reminder that
it's very easy to make mistakes when the meaning of an action is
overloaded to potentially mean two different things.

-Phil

> For instance, the REPL in Emacs does not send anything to
> Fennel until it has a complete expression.

In case of proto-repl it's because the protocol isn't designed for 
partial input. In case of a plain REPL, it's purely for editing 
convenience - being able to go back and edit previous line as in normal 
code buffer. I't also makes implementation a bit less involved, because 
comint is really hard to work with. That aside, nothing prevents plain 
REPL from sending incomplete expressions.

-- 
Andrey Listopadov

Andrey Listopadov <andreyorst@gmail.com> writes:

> That aside, nothing prevents plain REPL from sending incomplete
> expressions.

Yeah, I guess I was a bit unclear about it, but I meant it as a good
thing; Emacs is smart enough to not send incomplete forms. I think the
line-based chunking is useful in some contexts, but it's kind of
low-level, and if you don't need to rely on that behavior, it's probably
better not to.

-Phil