RFC SUMMARY

Some libraries export functions that operate on opaque pointers. In C,
this is convenient: in the header you declare the type without defining
it, then you define and use it as normal in the source file. In Hare,
this is not always convenient. You typically have two options: export
the type (and all types it uses, which may be Lots), or export an opaque
type and cast it to and from an internal type at your library
boundaries. These options work, but neither is ideal for large libraries
with many internal types.

I propose allowing type declarations to specify that the type is opaque
externally to the module, while remaining visible internally.

                         LANGUAGE IMPLICATIONS

Allow an optional "opaque" after "export" in type declarations, such as

	export opaque type context = struct {
		internal_thing: internal_type, // not exported!
		// ... and so on
	};

Within the module, the type can be used as normal. The only restriction
is that it can only be used in exported definitions in the same places
that opaque can be used. Outside of the module, the type is identical to
an opaque alias.

                     STANDARD LIBRARY IMPLICATIONS

None. The stdlib should typically not use this feature.

                         ECOSYSTEM IMPLICATIONS

This is useful for those libraries that would use opaque pointers in C.

For example, I am writing an optimizing compiler library with an
interface similar to libgccjit. If I go the route of exporting the main
structs (context, function, basic block), then over a hundred internal
types will also need exporting. Many of these have attached comments and
so would show up in haredoc, despite being purely internal. Better is
to export opaque structs and cast to and from an internal version.
However, this leads to a lot of ugly code, especially when internal
functions want to call exported functions, since now they need to cast
back to the opaque type. These casts should be free in terms of
runtime, but they clog up the source code.

                                  MISC

It might instead make sense to put the opaque after the equals sign,
making it a property of the type rather than the declaration.

On Mon, Dec 16, 2024 at 10:06:40PM +0000, Joe Finney wrote:
>                               RFC SUMMARY
> 
> Some libraries export functions that operate on opaque pointers. In C,
> this is convenient: in the header you declare the type without defining
> it, then you define and use it as normal in the source file. In Hare,
> this is not always convenient. You typically have two options: export
> the type (and all types it uses, which may be Lots), or export an opaque
> type and cast it to and from an internal type at your library
> boundaries. These options work, but neither is ideal for large libraries
> with many internal types.

-1 for me, just for the reason of simplicity; i don't think that
the benefits outweigh the complications, which would be a special
case in every documentation tool and i suspect in the compiler
aswell, if i am understanding this RFC correctly. admittedly, these
are quite minimal, but so is the problem that is solved. it's fine
to export the fields of structs or other stuff to the "outside
world", meaning outside of the module IMHO. also, it's pretty easy
to grep for stuff in documentation, given that you only have to
search for "type x" or "fn x" and you're straight at the definition.

and exporting types has some other benefits too, like beeing able to
allocate them on the stack.

I like it. I had thought about adding something to like this to the
language for a very long time. This is a lot cleaner than the convention
of not documenting opaque types to signal that it shouldn't be used.

The fact that the "opaque" keyword is already established is a blessing
and a curse; a blessing since I don't have to yell about adding new
keywords, and a curse because it's being overloaded a bit. But I think
on the balance of things the meanings are similar enough.

On Mon Dec 16, 2024 at 11:06 PM CET, Joe Finney wrote:
>                          LANGUAGE IMPLICATIONS
>
> Allow an optional "opaque" after "export" in type declarations, such as
>
> 	export opaque type context = struct {
> 		internal_thing: internal_type, // not exported!
> 		// ... and so on
> 	};
>
> Within the module, the type can be used as normal. The only restriction
> is that it can only be used in exported definitions in the same places
> that opaque can be used. Outside of the module, the type is identical to
> an opaque alias.

I have an implementation concern: we will probably have to export the
full type to the typedef file from the compiler, and the compiler will
have to know where a type comes from to enforce opaqueness. We need to
know its size and alignment at least for it to be, for instance,
allocated on the stack by an external callsite.

I also wonder if this should be permitted on other exported
declarations? e.g. export opaque fn. It would have no semantic
implications but could serve to hide functions/defs/etc from haredoc
which are exported for pragmatic reasons but should not ordinarily be
used.

>                      STANDARD LIBRARY IMPLICATIONS
>
> None. The stdlib should typically not use this feature.

On the contrary I can think of dozens of cases where the stdlib could
stand to utilize this feature.

>                                   MISC
>
> It might instead make sense to put the opaque after the equals sign,
> making it a property of the type rather than the declaration.

-1

Hi,

thanks for proposing this. The fact that Hare has no way to "hide" the
internals of a type from the outside world is indeed annoying at times,
and would probably turn out to be actively problematic if we had a more
active ecosystem of third-party libraries. This is one way to solve
this, another would be to have a way to "hide" member of a struct from
module consumers. Have you considered anything like that? It would be
more fine grained, but also probably more complicated to implement than
this one.

>                               RFC SUMMARY
>
> Some libraries export functions that operate on opaque pointers. In C,
> this is convenient: in the header you declare the type without defining
> it, then you define and use it as normal in the source file. In Hare,
> this is not always convenient. You typically have two options: export
> the type (and all types it uses, which may be Lots), or export an opaque
> type and cast it to and from an internal type at your library
> boundaries. These options work, but neither is ideal for large libraries
> with many internal types.
>
> I propose allowing type declarations to specify that the type is opaque
> externally to the module, while remaining visible internally.
>
>                          LANGUAGE IMPLICATIONS
>
> Allow an optional "opaque" after "export" in type declarations, such as
>
> 	export opaque type context = struct {
> 		internal_thing: internal_type, // not exported!
> 		// ... and so on
> 	};
>
> Within the module, the type can be used as normal. The only restriction
> is that it can only be used in exported definitions in the same places
> that opaque can be used. Outside of the module, the type is identical to
> an opaque alias.

How opaque are these types actually meant to be? Did you intend for them
to be stack-allocatable? Passable to functions by value? Allowed to be
members in other types?

I understand this rfc as if the answer is implictly no to all, but imo
allowing these would actually be a good idea and make them much more
useful. As Drew already said, that's not going to be trivial to
implement though.

> For example, I am writing an optimizing compiler library with an
> interface similar to libgccjit.

Sorry, offtopic, but this sounds very interesting :)

> It might instead make sense to put the opaque after the equals sign,
> making it a property of the type rather than the declaration.

I think it should be a property of the declaration. Otherwise you could,
syntactically speaking, specify do this for non-exported types and that
makes no sense and the check for that would be weird in the compiler.

On Tue Dec 17, 2024 at 8:57 PM CET, Bor Grošelj Simić wrote:
> Hi,
>
> thanks for proposing this. The fact that Hare has no way to "hide" the
> internals of a type from the outside world is indeed annoying at times,
> and would probably turn out to be actively problematic if we had a more
> active ecosystem of third-party libraries. This is one way to solve
> this, another would be to have a way to "hide" member of a struct from
> module consumers. Have you considered anything like that? It would be
> more fine grained, but also probably more complicated to implement than
> this one.

I should have mentioned this in my own feedback, but the idea of hiding
specific struct members had occured to me and I felt it was not worth
it. Just do the following:

export type some_struct = struct {
	public: int,
	members: int,
	go: int,
	here: int,
	private_members,
};

export opaque type private_members = struct {
	...
};

For full disclosure when I was initially mocking up this feature back
in, err, 2020 or so, I was pondering something like this:

export type example = struct {
	@private foo: int,
	// Visible to another specific module(s)
	@private(other::module, ...) bar: int
};

But I do not think this is a good idea. I like this RFC better.

> How opaque are these types actually meant to be? Did you intend for them
> to be stack-allocatable? Passable to functions by value? Allowed to be
> members in other types?
>
> I understand this rfc as if the answer is implictly no to all, but imo
> allowing these would actually be a good idea and make them much more
> useful. As Drew already said, that's not going to be trivial to
> implement though.

I don't think it's going to be *hard* to implement, though. Just
nontrivial. If it's opaque while compiling a file normally, it's more or
less a no-op, if it's opaque while loading typedefs, enforce it.

So, I like the idea of this, but its utility is somewhat limited. As you
mentioned, we wouldn't be able to use this in the stdlib for the most
part, since we return stack allocated objects. So we wouldn't be able to
get rid of the convention of not documenting internal types with this
RFC alone.

As I see it, the two big advantages this has over some way of marking
specific fields as private/internal is 1) not needing to export every
internal type used by a field (as you mentioned), and 2) enforcing that
a type's ABI isn't stable (by requiring it to be used indirectly). The
former would be useful in some places in the stdlib, such as regex::, if
we could somehow make it work with stack-allocated types. The latter is
specific to this proposal (and useful).

Here's an idea: how about dropping the requirement that a type
referenced indirectly by an exported type needs to also be exported?
That wouldn't require any additional syntax; we just make it so types
which don't *need* to be exported for other exported types to work don't
need to be. In this case, they'd be treated as opaque by other modules
(same as your proposal). This would keep the two above advantages of
your proposal. As for documentation: we could allow haredoc to view docs
for unexported declarations, if requested (sorta similar to godoc's -u,
but possibly without any command-line flag; just allowing unexported
docs to be requested individually. This would complicate the HTML docs
though).

If we do go with an explicit syntax, I'm kinda eh on overloading opaque
like this, but it's fine I guess.

On Mon Dec 16, 2024 at 5:06 PM EST, Joe Finney wrote:
>                                   MISC
>
> It might instead make sense to put the opaque after the equals sign,
> making it a property of the type rather than the declaration.

-1, for the reasons others already brought up.

On Tue Dec 17, 2024 at 3:02 PM EST, Drew DeVault wrote:
> For full disclosure when I was initially mocking up this feature back
> in, err, 2020 or so, I was pondering something like this:
>
> export type example = struct {
> 	@private foo: int,
> 	// Visible to another specific module(s)
> 	@private(other::module, ...) bar: int
> };
>
> But I do not think this is a good idea. I like this RFC better.

Annotations could also be used for this:

	#[internal] bar: int, // or #[lint::internal], whatever

This wouldn't be enforced by the compiler, but it could cause a warning,
or just be something a linter yells at you for. Then there could also be
an annotation to locally "suppress" the warning, so the declaration
itself doesn't need to list specific modules which are allowed to use
the field.

On Tue Dec 17, 2024 at 9:25 PM CET, Sebastian wrote:
> So, I like the idea of this, but its utility is somewhat limited. As you
> mentioned, we wouldn't be able to use this in the stdlib for the most
> part, since we return stack allocated objects. So we wouldn't be able to
> get rid of the convention of not documenting internal types with this
> RFC alone.

See my review regarding stack allocation, it would work fine. It
wouldn't behave exactly like opaque but it would be quite possible to
prevent users from field-accessing "opaque" types from external modules
while still knowing their size and alignment for stack allocation.

To be clear, how this would support stack allocations in practice is:

// foo::
export opaque type bar = struct { x: int, y: int };

// foo.td
export opaque type bar = struct { x: int, y: int };

Then when importing foo, harec notes that it's processing a typedef and
any opaque declarations are flagged as foreign opaque declarations in
that context. These are imported into the type store normally, so harec
understands how to allocate them and such, but are limited in the user's
ability to interact with them; namely only allowing assignment and
straightforward access expressions (but not, e.g. field access or
indexing).

> Here's an idea: how about dropping the requirement that a type
> referenced indirectly by an exported type needs to also be exported?
> That wouldn't require any additional syntax; we just make it so types
> which don't *need* to be exported for other exported types to work don't
> need to be. In this case, they'd be treated as opaque by other modules
> (same as your proposal). This would keep the two above advantages of
> your proposal. As for documentation: we could allow haredoc to view docs
> for unexported declarations, if requested (sorta similar to godoc's -u,
> but possibly without any command-line flag; just allowing unexported
> docs to be requested individually. This would complicate the HTML docs
> though).

This would probably be more complicated than the original RFC
as-written, since you would have to... export unexported types to the
typedef file? or something?

On Tue Dec 17, 2024 at 3:48 PM EST, Drew DeVault wrote:
> On Tue Dec 17, 2024 at 9:25 PM CET, Sebastian wrote:
> > So, I like the idea of this, but its utility is somewhat limited. As you
> > mentioned, we wouldn't be able to use this in the stdlib for the most
> > part, since we return stack allocated objects. So we wouldn't be able to
> > get rid of the convention of not documenting internal types with this
> > RFC alone.
>
> See my review regarding stack allocation, it would work fine. It
> wouldn't behave exactly like opaque but it would be quite possible to
> prevent users from field-accessing "opaque" types from external modules
> while still knowing their size and alignment for stack allocation.
>
> To be clear, how this would support stack allocations in practice is:
>
> // foo::
> export opaque type bar = struct { x: int, y: int };
>
> // foo.td
> export opaque type bar = struct { x: int, y: int };
>
> Then when importing foo, harec notes that it's processing a typedef and
> any opaque declarations are flagged as foreign opaque declarations in
> that context. These are imported into the type store normally, so harec
> understands how to allocate them and such, but are limited in the user's
> ability to interact with them; namely only allowing assignment and
> straightforward access expressions (but not, e.g. field access or
> indexing).

Hm, yeah makes sense.

I do think it might be good to allow only marking certain fields as
internal (or as exported), like what you mentioned in a previous email
(by subtyping an opaque struct into a non-opaque struct). But I'm also
not entirely convinced `opaque` is the correct way to go about it? I
guess it works though.

> > Here's an idea: how about dropping the requirement that a type
> > referenced indirectly by an exported type needs to also be exported?
> > That wouldn't require any additional syntax; we just make it so types
> > which don't *need* to be exported for other exported types to work don't
> > need to be. In this case, they'd be treated as opaque by other modules
> > (same as your proposal). This would keep the two above advantages of
> > your proposal. As for documentation: we could allow haredoc to view docs
> > for unexported declarations, if requested (sorta similar to godoc's -u,
> > but possibly without any command-line flag; just allowing unexported
> > docs to be requested individually. This would complicate the HTML docs
> > though).
>
> This would probably be more complicated than the original RFC
> as-written, since you would have to... export unexported types to the
> typedef file? or something?

We already have logic in harec to check if an exported type references
an unexported type, so I can't imagine it'd be much more difficult to
conditionally emit `export type module::T = opaque;` to the typedef file
when necessary.

And won't we need to figure out a way to exchange unexported
declarations between modules anyway for @inline?

On Tue Dec 17, 2024 at 10:20 PM CET, Sebastian wrote:
> I do think it might be good to allow only marking certain fields as
> internal (or as exported), like what you mentioned in a previous email
> (by subtyping an opaque struct into a non-opaque struct). But I'm also
> not entirely convinced `opaque` is the correct way to go about it? I
> guess it works though.

I think this RFC can proceed without answering this question, can defer
to later RFC.

> We already have logic in harec to check if an exported type references
> an unexported type, so I can't imagine it'd be much more difficult to
> conditionally emit `export type module::T = opaque;` to the typedef file
> when necessary.

Still need to know it's size etc if it's for example included in an
exported non-opaque struct as a value (and not a reference/pointer).

> And won't we need to figure out a way to exchange unexported
> declarations between modules anyway for @inline?

Yeah, eventually, but does not need to concern this RFC.

On 17.12.24 21:30, Sebastian wrote:
> On Tue Dec 17, 2024 at 3:02 PM EST, Drew DeVault wrote:
>> For full disclosure when I was initially mocking up this feature back
>> in, err, 2020 or so, I was pondering something like this:
>>
>> export type example = struct {
>> 	@private foo: int,
>> 	// Visible to another specific module(s)
>> 	@private(other::module, ...) bar: int
>> };
>>
>> But I do not think this is a good idea. I like this RFC better.
> 
> Annotations could also be used for this:
> 
> 	#[internal] bar: int, // or #[lint::internal], whatever

I think it's better to proactively export fields and make internal 
fields as default. In my work on hare the big majority of fields are 
internal. If you forget to internalize a field, you won't notice until 
the field is used somewhere and you want to change the field during 
refactor. This is even more inconvenienced when someone else is 
dependent on your library and uses the field. If you forget to 
externalize a field, you'll notice it more easily when you need to 
access it outside and it's also simpler to fix. Users should think about 
what fields should be externalized, not the other way around.

I think it can work somewhat with the nested opaque struct Drew 
mentioned, but you still need to remember to start structs as opaque, if 
you intend to have internal fields.

As a side note: I like the idea of explicitly marking a field as 
exported. Because if you take go as an example: If you want to 
externalize a field later on, you have to change the name and thus 
refactor all the dependent code, which is annoying and may just lead to 
a bad habit to always externalize a field by default.

My initial proposal was to make the types properly opaque, meaning they
could not be stack allocated, passed by value, etc. This would suffice
for my use case, and is also super easy to implement: just literally put
"type t = opaque" in the .td file and you're most of the way there.

It seems the rough consensus is that we want to be able to stack
allocate, which is fair, although the code will be more complicated.

What do you expect would go into the .td file for

	type foo = int;
	export opaque type bar = struct { x: int, y: foo };

? Would it be acceptable to "export opaque" all types within a module,
or maybe just those types that are referenced in exported definitions,
so it would be

	// .td
	export opaque type foo = int;
	export opaque type bar = struct { x: int, y: foo };

? This seems suboptimal, because now users can use the name ::foo,
even if only as an opaque alias. Maybe put a bare "type foo = int" in
the .td file?

I can start playing around with options and get a patch out relatively
soon, but would love to hear if there's an obvious way to go.

On Wed Dec 18, 2024 at 12:57 PM CET, Joe Finney wrote:
> My initial proposal was to make the types properly opaque, meaning they
> could not be stack allocated, passed by value, etc. This would suffice
> for my use case, and is also super easy to implement: just literally put
> "type t = opaque" in the .td file and you're most of the way there.
>
> It seems the rough consensus is that we want to be able to stack
> allocate, which is fair, although the code will be more complicated.
>
> What do you expect would go into the .td file for
>
> 	type foo = int;
> 	export opaque type bar = struct { x: int, y: foo };

I'm okay with making users export foo as well for now. We can think up
alternatives later.

I think it makes the most sense to allow for non-exported decls in the
.td file. Such decls will not be directly usable by the module that
imports them, but they can be used by opaque types within their own
module. So,

	// foo/foo.ha
	type foo = int;
	export type bar = struct { x: foo };

	// foo.td
	type foo::foo = int;
	export type foo::bar = struct { x: foo::foo };

While importing the .td file, foo::foo is usable by foo::bar, but
foo::foo does not end up in the final scope returned from the import (I
think that's probably the best way?) and so cannot be used by the
importer.


If we only emit types that are referenced by exported types, then the
.td file will not grow at all, because those types would have needed
exporting regardless. It would likely also be fine to unconditionally
emit all types to the .td file, because parsing them is a relatively
small part of overall hare compilation (qbe dominates most programs).


> I also wonder if this should be permitted on other exported
> declarations? e.g. export opaque fn. It would have no semantic
> implications but could serve to hide functions/defs/etc from haredoc
> which are exported for pragmatic reasons but should not ordinarily be
> used.

Functions and globals could sensibly be opaque. They could use
unexported types, and they would not be callable/settable. I could
imagine a library that exposes various functions that are not meant to
be called by the user, but instead passed by pointer to some other
functions in the library. Then it would make some sense, and likewise
for globals. I don't think opaque defs make sense, because no storage.

quick drive-by comment: given that we're (iiuc) planning to stick the 
internals of exported-opaque types in typedef files, i'd like to have 
some sort of escape hatch for external accesses to exported-opaque 
types. go doesn't have a way to do this, and it's caused me a fair bit 
of frustration in the past while eg. debugging or hacking something 
together

You can always copy/paste the type and do a pointer cast, I guess. Do
you have a particular syntax/semantics in mind, and are you okay with
export opaque going in before this is implemented?

On Sat Jan 4, 2025 at 4:25 PM UTC, Joe Finney wrote:
> You can always copy/paste the type and do a pointer cast, I guess.

eh, yeah, that's good enough. +1