There are lots of ways to maintain hidden definitions among the ones
that are going to be exported from the module. As a mainly library author
I'm interested in an additional compile-time check of that.  I know that the
linter plugin can detect unused definitions, and the proposal I'm describing
here falls into a similar category, though maybe it should be on by default
like the strict global checking.

Currently, one can write all functions using regular (fn name []) syntax,
and select which definitions are going to be exported by adding those into
a (sometimes) giant table at the bottom of the module.  For big enough
libraries, this table becomes hard to navigate, and some functions can be
missed, or some functions that were meant for internal only use can leak
to the public API unnoticed. The linter can help with the first problem, but
not with the last one.

One solution is to define an 'ns' table at the top of the file and use the
(fn ns.name []) syntax, exporting the  'ns' table at the end of the module. 
This works, and public functions can be easily distinguished from private
ones, but makes it tedious to reuse public functions in the same file,
because of now required table lookup, that often means introducing
another local with a similar name for tight loop optimization.  I've tried
fixing this with a custom fn macro, but it introduces other problems.

I would like to propose a small addition to the inbuilt set of specials, that
have no differences in runtime semantics of the generated code, and only
meant to provide compile-time checks: fn-, lambda-, local-, and var-.

Defining a function with the (fn- name []) syntax will work exactly the
same as with the regular fn, with the exception of extra compile-time
check, that the name doesn't appear in the exported table literal.  Same
goes for local- and var-.  Additionally, if the module exports a sole function,
or a table via its name, a check can be performed that the name wasn't
declared with the fn- or local-.

I was using a similar method in some of my projects, defining functions
as (fn -name []) and while it makes it easier to spot such functions in
the exported table, using them feels kinda unpleasant: (-name ...).
A static check would be both more robust and make code easier to read.

As with everything compile-time based, this check can't be performed
withouts deep code analysis if the module is exported via the named local,
or if the exported table has nested tables.  I think it's OK though, some
safety is better than none at all.  Additionally, fn- may disallow the
multisym syntax for defining functions, preventing accidental exporting
as a part of some other table object.

Let me know what you think!
-- 
Andrey Listopadov

Andrey <andreyorst@gmail.com> writes:

> Let me know what you think!

An alternative to the proposed improvement:

(1) 'fn [TABLE].SYMBOL ...' defines a normal function, as it does now,

and

(2) 'fn+ SYMBOL ...' defines a function that the compiler puts into an
automatically synthesized export table at the end of the generated Lua
file.

P.S. 'fn+' is a placeholder name; it could be 'pub/fn' or whatever.

Rudy
-- 
"Mathematics takes us still further from what is human into the region
of absolute necessity, to which not only the actual world, but every
possible world, must conform."
-- Bertrand Russell, 1902

Rudolf Adamkovič <salutis@me.com> [he/him]
Studenohorská 25
84103 Bratislava
Slovakia

> 'fn+ SYMBOL ...' defines a function that the compiler puts into an
>automatically synthesized export table at the end of the generated Lua
>file.

Fennel tries to do as little implicit stuff as possible so this is out of the scope
I believe.  The static check doesn't produce any implicit behavior or altering
resulting code, and that's a big difference.

-- 
Andrey Listopadov

Andrey <andreyorst@gmail.com> writes:

> Currently, one can write all functions using regular (fn name []) syntax,
> and select which definitions are going to be exported by adding those into
> a (sometimes) giant table at the bottom of the module.  For big enough
> libraries, this table becomes hard to navigate, and some functions can be
> missed, or some functions that were meant for internal only use can leak
> to the public API unnoticed. The linter can help with the first problem, but
> not with the last one.

Hm; I can't shake the feeling that the problem you're describing is a
problem for a linter to solve. Now that the `fn` macro accepts arbitrary
metadata tables, the linter could be extended to check for this:

    (fn myfunction [x y z] {:private true}
      ...)

> Defining a function with the (fn- name []) syntax will work exactly the
> same as with the regular fn, with the exception of extra compile-time
> check, that the name doesn't appear in the exported table literal.

I feel like having `fn` and `fn-` where the latter is private seems to
imply that `fn` will declare a public function, but this is misleading
to people who are used to other languages' module systems. The metadata 
approach makes it clearer that the intent is for linting. It just feels
like a pretty big change for a use case which is comparatively obscure.

-Phil

> Hm; I can't shake the feeling that the problem you're describing is a
> problem for a linter to solve. Now that the `fn` macro accepts arbitrary
> metadata tables, the linter could be extended to check for this:
>
>     (fn myfunction [x y z] {:private true}
>       ...)

True.  We can't, however, add metadata to arbitrary locals.  Which is
not uncommon to have a local storing up a singleton that is used
internally and shouldn't bother users at all.  But I agree, metadata
solves this case for the most part.

On that note, should we start incorporating such metadata in Fennel
compiler itself?  I'd also like to see a :deprecated metadata be used,
as it will allow marking such functions in a special way in
auto-completion feature of proto-repl so they stand out more.

>     (fn myfunction [x y z] {:private true}
>       ...)

Also, a bit unrelated note but I think we should expand our metadata
ingerface a bit more, and allow both metadata table and the docstring as
separate items in the function body.  E.g.:

(fn myfunction [x y ...]
  "Docs go here.
Also multiline docs look much nicer."
  {:fnl/arglist [x y & rest]}
  (let [rest [...]]
    ;; body
    ))

It is less cumbersome to write, and the editor can highlight docstring
properly, unlike to when it is in the map:

(fn myfunction [x y ...]
  {:fnl/docstring "Docs go here.
Also multiline docs look much nicer."
   :fnl/arglist [x y & rest]}
  (let [rest [...]]
    ;; body
    ))

I've been writing a lot of functions with metadata tables and docs
always look out of place in the table.  What do you think?

--
Andrey Listopadov

Andrey Listopadov <andreyorst@gmail.com> writes:

> Also, a bit unrelated note but I think we should expand our metadata
> ingerface a bit more, and allow both metadata table and the docstring as
> separate items in the function body.  E.g.:
>
> (fn myfunction [x y ...]
>   "Docs go here.
> Also multiline docs look much nicer."
>   {:fnl/arglist [x y & rest]}
>   (let [rest [...]]
>     ;; body
>     ))

Yes, I think that's a good idea.

> On that note, should we start incorporating such metadata in Fennel
> compiler itself?  I'd also like to see a :deprecated metadata be used,
> as it will allow marking such functions in a special way in
> auto-completion feature of proto-repl so they stand out more.

Yes, we can start emitting warnings when calls to deprecated forms are
used. But I think first we need to offer a little more control around
these warnings; right now I don't even think we have a way to suppress
them.

-Phil

> Yes, I think that's a good idea.

Will send a patch soon.

>> On that note, should we start incorporating such metadata in Fennel
>> compiler itself?  I'd also like to see a :deprecated metadata be used,
>> as it will allow marking such functions in a special way in
>> auto-completion feature of proto-repl so they stand out more.
>
> Yes, we can start emitting warnings when calls to deprecated forms are
> used. But I think first we need to offer a little more control around
> these warnings; right now I don't even think we have a way to suppress
> them.

I think we shouldn't emit warnings on calls, only when we detect uage of
a deprecated function at compile time.  Runtime warnings are not
standardized across Lua, e.g. Lua 5.4 has a special `warn` function that
emits a warning, whereas older should rely on `io.stderr:write` which
may lead to problems if `warn` allows customization.

Emitting warnings at compile time is fine, because Fennel code is always
compiled.

Andrey Listopadov <andreyorst@gmail.com> writes:

>> Yes, we can start emitting warnings when calls to deprecated forms are
>> used. But I think first we need to offer a little more control around
>> these warnings; right now I don't even think we have a way to suppress
>> them.
>
> I think we shouldn't emit warnings on calls, only when we detect uage of
> a deprecated function at compile time.

Oh, haha, yeah; that's what I meant by "used" but I was unclear. I don't
think runtime warnings are a good idea.

-Phil