Hi,

I had some time so I tried to replace my system pkg-config (pkgconf
actually) with "u-config generic" to see how it goes. Here's some
observations:

Before installing u-config system-wide, I was comparing some u-config vs
pkgconf output and noticed that the `--cflags` output of u-config and
pkgconf seems to differ quite a lot:

	[~]~> u-config --cflags xft
	-I/usr/include
	-I/usr/include
	[~]~> pkgconf --cflags xft
	-I/usr/include/freetype2 -I/usr/include/harfbuzz -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include -I/usr/include/uuid 

Not entirely sure why this is. But I tried out u-config on two
applications that require xft and it went fine.

After this, I patched the generic_main.c to log every invocation of it
and output it to `/var/log/u-config`. And then installed that system-wide.

	+    FILE *f = fopen("/var/log/u-config", "a");
	+    if (!f) abort();
	+    char *pwd = getcwd(NULL, 0);
	+    fprintf(f, "[package]: %s\n", pwd ?: "UNKNOWN");
	+    fputc('\t', f);
	+    for (int i = 0; i < argc; ++i) {
	+        fprintf(f, " %s", argv[i]);
	+    }
	+    fputc('\n', f);
	+    fflush(f);
	+
	     appmain(conf);
	 
	+    if (!ferror(stdout)) {
	+        fprintf(f, "\t--- success (maybe) ---\n");
	+    }
	+    fclose(f);

The first thing I noticed in the logs was that there was two pkg-config
calls when installing /u-config itself/, which tells me that portage is
the one who's calling it:

	 --variable=completionsdir bash-completion
	 --exists systemd

Not sure why it's asking pkg-config for the existence of `systemd` when
portage *itself* should know that already. But it was causing a build
failure so I added a dirty patch for `--exists`:

	--- a/u-config.c
	+++ b/u-config.c
	@@ -1525,6 +1525,7 @@ static void appmain(Config conf)
	     Bool libs = 0;
	     Bool cflags = 0;
	     Bool modversion = 0;
	+    Bool exists = 0;
	     Str variable = {0, 0};
	 
	     proc.define_prefix = conf.define_prefix;
	@@ -1559,6 +1560,9 @@ static void appmain(Config conf)
	             flush(&out);
	             return;
	 
	+        } else if (equals(r.arg, S("-exists"))) {
	+            exists = 1;
	+
	         } else if (equals(r.arg, S("-modversion"))) {
	             modversion = 1;
	 
	@@ -1645,6 +1649,10 @@ static void appmain(Config conf)
	         os_fail();
	     }
	 
	+    if (exists) {
	+        return; /* FIXME: this is kinda hacky... */
	+    }
	+
	     if (modversion) {

After this the next thing caught by the log are these:

	[package]: sys-libs/libseccomp-2.5.4
		 --maximum-traverse-depth=1 --with-path=/var/tmp/portage/sys-libs/libseccomp-2.5.4/image/usr/lib*/pkgconfig --with-path=/var/tmp/portage/sys-libs/libseccomp-2.5.4/image/usr/share/pkgconfig --validate /var/tmp/portage/sys-libs/libseccomp-2.5.4/image/usr/lib64/pkgconfig/libseccomp.pc
	[package]: sys-apps/file-5.44-r3
		 --maximum-traverse-depth=1 --with-path=/var/tmp/portage/sys-apps/file-5.44-r3/image/usr/lib*/pkgconfig --with-path=/var/tmp/portage/sys-apps/file-5.44-r3/image/usr/share/pkgconfig --validate /var/tmp/portage/sys-apps/file-5.44-r3/image/usr/lib64/pkgconfig/libmagic.pc

Looks like it's attempting to "validate" the .pc files.  I'm suspecting
portage is the one making these calls too, but I'm not sure. But in any
case, whoever is making call doesn't seem to be checking the return code
because the build succeeded anyways (despite u-config not supporting
almost any of those flags).

The final boss (for now) was autohell. I'll spare you the details about
how the python autoconfig fails to find pkg-config but goes ahead and
starts the build anyways just to fail mid-way through - but the
important part here is that pkgconf provides some sort of "pkg.m4" which
python (and imlib2) autoconfig files seem to depend on.

https://gitea.treehouse.systems/ariadne/pkgconf/src/branch/master/pkg.m4

At this point I decided to call it a day since there's (unfortunately)
too many packages that depend on autohell (either directly or indirectly
via one of it's dependencies).

To summarize, it appears that more work has to be done before u-config
can be a viable replacement for pkg-config system-wide.

1. It most likely will need to support `--exists`.
2. It most likely will need to provide this "pkg.m4" file.
3. Perhaps it might need to ignore `--validate` flag (i.e exit with 0)
   if people finally start checking the return code.

- NRK

On Fri, Jan 20, 2023 at 08:31:21PM +0600, NRK wrote:
> Before installing u-config system-wide, I was comparing some u-config vs
> pkgconf output and noticed that the `--cflags` output of u-config and
> pkgconf seems to differ quite a lot:

I meant to say for `xft` specifically. For most packages, the output is
more or less equal.

Also, one small typo in the README:

	-pkg-config's idiosynratic argument parsing — positional arguments are
	+pkg-config's idiosyncratic argument parsing — positional arguments are^M

- NRK

Thanks for catching this! Turns out I had misunderstood the interaction 
between Requires.private and --cflags. It should output private cflags 
since these will be necessary for the includes and such to work. Private 
--libs are excluded when dynamic linking to avoid "overlinking," and it 
instead relies on the dependency information already present in dynamic 
libraries.

I've addressed this in 269cbb4, which now outputs these private cflags. I 
needed extra bookkeeping for this, but it's cleaner overall. Since the 
dependency tree is flattened during the load, it may be possible that in a 
deep dependency tree a package is loaded privately, then later needed 
publicly, but it's left private when it should be public, but I was unable 
to contrive a case where this actually happens.

By the way, virtualizing the file system via os_mapfile is going to be a 
boon when building a test suite. I can construct virtual pkgconfig/ 
directories with as many packages as needed without actually writing a 
single file. Great for testing contrived cases of the above. (I haven't 
built a test suite yet because, as shown here, I'm still working out the 
correct behavior.)

Downside: xft in particular now outputs a ton of redundant arguments, so 
I'm going to address that issue after all. Should be easy.

I'm a bit surprised the builds worked with the old --cflags output. That 
suggests these programs include headers using the wrong paths, like using 
"SDL2/SDL.h". It usually works out by accident so nobody notices. That is, 
until someone comes along wanting to link against a custom build of the 
library, and the system headers are incompatible with the custom build, 
resulting in a broken build! (This is common enough that SQLite provides a 
way to defend against such build mistakes.)

Great idea on the trace! I'm glad you were able to hack it in. I probably 
would have used a wrapper script.

    #!/bin/sh
    echo $(pwd) "$@" >>/var/log/u-config
    exec real-pkg-config "$@"

Another idea:

Finding people's use of /usr/bin/python with the Linux audit framework
https://utcc.utoronto.ca/~cks/space/blog/linux/FindingPython2UsesWithAudit

As for --exists, since it's used in practice I'll add it. It just disables 
standard error, even in the original pkg-config. The default behavior is 
already to test the arguments.

I had a --validate stub that always succeeded, but removed it. (As stated 
in the README, the goal is merely to support existing builds, not help 
write new .pc files.) I might wait to restore it since you didn't actually 
need it, plus it would additionally require supporting (or also ignoring) 
those pkgconf-only arguments (--with-path, etc.), since it fails on those 
first. I'll think on this.

> pkgconf provides some sort of "pkg.m4"

Gross. According to Debian apt-file, both pkg-config and pkgconf provide 
it, and they're the same file. IMHO, this really ought to be a part of 
autotools itself. Fortunately for me this is out of scope. :-) The person 
packaging u-config for a distribution is responsible for including a copy 
of pkg.m4 in their package, especially since only they know where to put 
it (i.e. it depends on how the distribution packages autotools).

On 1/20/23 1:38 PM, Christopher Wellons wrote:
> Gross. According to Debian apt-file, both pkg-config and pkgconf provide
> it, and they're the same file. IMHO, this really ought to be a part of
> autotools itself. Fortunately for me this is out of scope. :-) The
> person packaging u-config for a distribution is responsible for
> including a copy of pkg.m4 in their package, especially since only they
> know where to put it (i.e. it depends on how the distribution packages
> autotools).


While it's true that automake should provide this, I suspect the
automake stance is simple and boils down to "autotools provides
primitives and doesn't know anything about pkg-config, so pkg.m4 should
be distributed with whatever provides the tool".

I guess you could try submitting an automake bug report?

But either way, it's totally invalid to pretend your hands are tied and
you "can't" distribute one because "you don't know where to put it".

The rule of where to put it is extremely simple, it must be installed
into whatever --datadir automake was compiled with, inside the aclocal/
subdirectory. So, assuming a distribution packages it as
--prefix=/usr/share, you must install to /usr/share/aclocal/ and it's
that simple. A reasonably standard assumption is that both tools are
going to be installed to the same location, which means installing to
`$(DATADIR)/aclocal/`.

If you're still not confident about where to install it (which seems odd
to me since the u-config Makefile does not seem to provide an install
rule? why are you concerned about "where to put it" -- it should be
sufficient to simply include it as a data file in git) then aclocal
guarantees that `aclocal --print-ac-dir` will print the directory you
should install these files into.


-- 
Eli Schwartz

Hi,

A small update on the situation: after the recent commits adding dummy
autoconf related flags (plus dropping "pkg.m4" into the right place),
imlib2 and python's autoconf script seems to be happy, at least when I
run them locally.

However, it fails when I try to build via portage:

	checking pkg-config is at least version 0.9.0... Aborted
	no

I don't see anything on my invocation log, so I suspect it's some env
related issue. Haven't had much time to investigate this further (and it
appears I won't have much time tomorrow either).

But in the little time I did have, I noticed that the python ebuild
calls pkg-config with `--keep-system-libs` when cross-compiling. Not an
issue for me since I don't cross-compile, but since it's used in
practice I figured you'd want to add this.

And I also figured it'd be easy to add after the recent duplicate
removal change. And indeed it was, patch attached (I didn't adjust the
windows layer though, as I don't have any means to test it).

Also noticed one more anomaly, haven't investigated it but I'd guess
it's related to the recent `.private` change:

	~> pkg-config --libs xft 
	-lXft 
	~> ./u-config --libs xft
	-lXft -lX11 -lfreetype

Finally, I wanted to correct something about my last post. It turns out
that `--validate` was indeed being called by portage, however the build
doesn't fail because it's a non-fatal QA check. I didn't notice the QA
warning on my build log the other day. So the result is actually being
checked (which it still fails due to the `--with-path` and
`--maximum-traverse-depth` flags).

- NRK

> why are you concerned about "where to put it"

The point is that I'm not. As integration between independent tools — 
taking your word at "provides primitives" — it can only by solved by 
distributions providing both intending them to work together. So I leave 
it to packagers to solve appropriately in their distribution, and they'll 
choose a pkg.m4 appropriate for it. A pkg-config shouldn't be concerned 
with pkg.m4 any more than it is with pkg.el, pkg.py, pkg.lua, or other 
integrations. Historically Freedesktop.org pkg-config distributed a 
pkg.m4, which pkgconf copied (literally, and that's not a bad thing), and 
it's why it's still packaged that way. However, it's probably a mistake 
for distributions to reflect this in their package structure today.

Case in point: it's definitely a mistake for Debian and its derivatives. I 
cannot install pkg-config and pkgconf simultaneously *only* because they 
conflict on pkg.m4 despite it being the same script. If that file was 
provided by another package, such as through Automake or even its own 
package, then there would be no conflict. I could install both and choose 
the provider of the "pkg-config" command (and man page) through the 
alternatives system. In this case the bug report would go to Debian, not 
Automake.

I've been testing as well, except building directly from source tarballs, 
which is why I'd added those accept-and-ignore options. The only case I 
have that still doesn't work is Python. That's because the libffi.pc in 
Debian Bullseye has the wrong information. It works with pkg-config and 
pkgconf because they exclude the wrong information by chance, and the 
fallback happens to work out anyway.

Unlike Autotools, portage exercises a lot of pkgconf-only features! I'm 
not running into these in my source tarball tests.

> checking pkg-config is at least version 0.9.0... Aborted

I suspect it's parsing the output of --version instead telling it to check 
itself with --atleast-pkgconfig-version. I noticed that pkgconf --version 
doesn't report that it's actually pkgconf, which is unusual. I suspected 
it's for compatibility. The first u-config "release" will be marked at 
least as high as 0.20.0 to avoid this issue. You could test this by 
changing the VERSION macro at the very top of u-config.c.

> --keep-system-libs 

I just added this and --keep-system-cflags as accepted and ignored, though 
now I just noticed your patch already deals with it. (I should have waited 
until I read your message in full!)

Has keeping the system paths actually hurt anything in your tests? I had 
planned to skip this since it didn't hurt as far as I could tell. (Though 
it *is* this feature that allows the broken libffi.pc to work out.)

> related to the recent `.private` change

Same on my system. I'll look into this.

> `--with-path` and `--maximum-traverse-depth`

I'll probably just implement these. For the first, I'll turn the search 
path into a linked list (fast/easy to prepend or append) which would make 
this trivial. There's already a maximum traverse depth, but I can hook it 
up to a command line argument, which will be min(default, arg).

I applied your patch, but with edits:

I didn't like that rem.s overflows in some cases: If dlen == rem.len, then 
"rem.s += dlen + 1" goes too far. I didn't want this to exist in any 
commit, even if immediately fixed, so that's when I decided to edit the 
patch. I first made the cut() default more sensible (b7fafa6) then applied 
it here.

My mindset has been that subscripts and especially manipulating "len" is 
"low level" and that "high level" functions should be preferred since 
they're easier to get right and are guarded by assertions. The needed high 
level functions have been gradually revealed as the program evolves — when 
I notice low level patterns and pull them up. I still need to revisit some 
of the older code and replace low level manipulations with cut() and 
friends.

Also, I "free" the string allocation if it wasn't needed. :-)

I turned insertsyspath into more of a "method" for OutConfig since that's 
ultimately what it's manipulating.

I forgot to mention: Do you know how to extract pkg-config's embedded 
system include/lib paths? I'd like to do the same as with pc_path, 
automatically configuring from the system's pkg-config. Matching its 
configuration has been useful for testing, but I know of no reliable way 
to get pkg-config to divulge it.

On Sun, Jan 22, 2023 at 04:27:19PM -0500, Christopher Wellons wrote:
> I didn't want this to exist in any commit, even if immediately fixed,

I try to do the same, so I know where you're coming from :)

> so that's when I decided to edit the patch. I first made the cut()
> default more sensible (b7fafa6) then applied it here.
> 
> My mindset has been that subscripts and especially manipulating "len" is
> "low level" and that "high level" functions should be preferred since
> they're easier to get right and are guarded by assertions.

While writing the patch I was looking for something like this and found
`splitpath`, but it's interface wasn't appropriate for what I was doing.
If I had noticed `cut` and `copy` then I'd probably have done something
similar. So the edit is fine by me!

> Unlike Autotools, portage exercises a lot of pkgconf-only features!

Presumably this is because the freedesktop pkg-config has been pruned
from the gentoo tree for a while now, making pkgconf the only available
pkg-config provider.

> I suspect it's parsing the output of --version

In my logs, I don't see _any_ invocation at all. Which was the weird
part. I'll have the decent amount of free time tomorrow, so hopefully
I'll figure it out :)

> Has keeping the system paths actually hurt anything in your tests?
> planned to skip this since it didn't hurt as far as I could tell. (Though it
> *is* this feature that allows the broken libffi.pc to work out.)

Due to the autoconf issue, I've only gotten to test about 7~9 packages
thus far. So it's not a large enough sample size to be drawing
conclusions from. But with that being said, I haven't noticed any issue
(aside from the python ffi issue you mention).

> I forgot to mention: Do you know how to extract pkg-config's embedded system
> include/lib paths?

For pkgconf, the following works (I think I learnt it accidentally due
to zsh auto-completion):

	[~]~> pkgconf pkg-config --variable pc_system_includedirs
	/usr/include
	[~]~> pkgconf pkg-config --variable pc_system_libdirs
	/lib64:/usr/lib64

No clue weather this works on the original pkg-config or not, I'd assume
not since this is apparently a pkgconf specific extension.

Also one more thing I've been meaning to ask, any specific reason for
not using `-ffreestanding` for the windows build if the plan is to not
link against libc?

- NRK

> something like this and found `splitpath`

That was one of those functions needing revisiting, and I indeed rewrote 
it yesterday in terms of cut() in order to support --with-path.

> pkgconf pkg-config --variable pc_system_libdirs

I should have realized that since your patch set these! I didn't know you 
were copying a pkgconf-only feature. So unfortunately it won't work if the 
system has pkg-config (i.e. still the default for Debian and derivatives).

The Debian package (in debian/rules) uses dpkg-architecture alongside "gcc 
-print-multi-lib" to construct its system library path, which is hardcoded 
as a string in the binary with no way to access it directly. The include 
path is left to the default.

At least for testing — since it's been so useful to test with an identical 
configuration — I wonder if I can rig up "strings" and "grep" to grab it 
reliably enough if pc_system_libdirs fails (i.e. it's not pkgconf)…

> any specific reason for not using `-ffreestanding`

This is a subject about I need to write another article, but some of the 
pieces are still unclear and I'm still figuring it out. Here's what I know 
right now.

Despite my earlier writing on the subject, I learned that -ffreestanding 
is not the appropriate option for CRT-free Windows builds (nor libc-free 
Linux builds for that matter). It's for kernels and such, and even then it 
only barely works. The GCC documentation says it's for an environment "in 
which the standard library may not exist" but if you follow through into 
the documentation for the implied -fno-builtin it says it's going to call 
the standard library anyway and you can't do anything about it. It's up to 
you to supply (at least part of) that standard library that's not supposed 
to exist.

Also, it's worse than the documentation suggests. It says it may generate 
calls to memcmp, memset, memcpy, and memmove, but that's not an exhaustive 
list. As seen with u-config, it also includes at least strlen.

The name -fno-builtin sounds a lot like the solution, but it's not. (Its 
documentation is really confusing.) It even appears to work most of the 
time, usually eliminating the libc calls, but it's really generating worse 
code and not solving the underlying issue. Using -ffreestanding just gives 
you -fbuilt-in and undefines __STDC_HOSTED__. The former is bad but the 
latter of which *is* kind of useful.

The situation is irritating, and honestly, GCC is doing a lousy job in 
this area. I suspect the GCC maintainers don't want to officially support 
it even though it's sometimes required (Linux kernel, libc itself), so 
they just don't make promises and instead expect those communities to keep 
abreast of GCC changes. Microsoft is no different about this with their 
own toolchain, and like GCC, they make it difficult as an unsupported 
configuration. See here:

https://hero.handmade.network/forums/code-discussion/t/94-guide_-_how_to_avoid_c_c++_runtime_on_windows

One difference compared to Mingw-w64 is that if you don't reference libc, 
MSVC won't link it. I take advantage to suggest super simple builds, but 
in other contexts it can lead to confusingly broken builds, when people 
are counting on the CRT to set things up, but MSVC thinks they don't need 
it.

The trickiest part of all is when GCC generates one of these libc calls 
you need to supply that function in a way that GCC won't optimize into 
infinite recursion. Options include:

1. Compile in a separate translation unit at a lower optimization level
(i.e. without the GCC 4.8+ -ftree-loop-distribute-patterns)

2. Disable that tree loop optimization using a function attribute

3. Implement using inline assembly 

4. Implement using arch-specific intrinsics

5. Stymy optimization using volatile or empty inline assembly

I've been using 4 and 3, in that order of preference. (Beware: most of the 
inline assembly samples you'll find online for these functions is WRONG!) 
At least on Windows there are only two architectures to worry about, x86 
and x64. (Yes, I realize there's technically ARM, too, but the support, 
tooling, and availability is poor.) It's easy enough to polyglot in 
assembly. Using (1) is inconvenient and complicates building, and I worry 
about the long-term prospects of (2). Since performance isn't an issue for 
u-config, I may ultimately end up doing (5).

I'm still figuring out all the details and adjusting as I learn. If you 
search the internet for information, you'll quickly bottom out on my own 
writing in various places (my blog, reddit, etc.). It would seem I'm the 
only person in the world actually building normal software like this, and 
everyone else who might want to do so quickly gives up and accepts CRT 
linkage.

A recent trick I learned working on u-config: -nostartfiles is friendlier 
than -nostdlib, and I'm going to suggest it more often. It still passes 
-lmingw32 and -lkernel32 to the linker, but doesn't try to take the entry 
point. If you don't reference the CRT, the linker will exclude it. This 
has two benefits: (1) don't need to remember system DLLs, and (2) if GCC 
generates an unanticipated libc call in the future, it will automatically 
link MSVCRT to supply it as a useful fallback. It's much the MSVC default.

I elaborated on more Windows-specific details in w64devkit commit 7950ba3:
https://github.com/skeeto/w64devkit/commit/7950ba3

Also check out my experimental "crt" and "nocrt" configurations in my 
.gdbinit for adapting to each situation. I'm not happy with it (note: 
Python is unsupported in native Windows GDB, which is mainly CPython's 
fault), but it's the best I've figured out so far. GDB doesn't break on 
abort() on Windows, and it's not designed to carve out special exceptions 
like abort(). (I've looked into it!) On other platforms, GDB relies on 
signals to trap abort, which Windows of course doesn't have, and MSVCRT 
abort() just exits (BOO!). This is part of why I like my simple ASSERT 
macro, though it also works better (in debuggers) than libc assert() on 
Linux (see: my article on this).

https://github.com/skeeto/dotfiles/blob/master/_gdbinit

The "pkg-config-linux-amd64" target is more experiment than serious, for 
me to try/test the same libc-free approach on Linux with a substantial, 
real world program. It has more inline assembly than I'd like. For 
learning purposes, I ported it to x32 (note: not i686), since it only 
requires a couple tweaks to make it work, but I didn't keep it.

On 1/22/23 1:46 PM, Christopher Wellons wrote:
>> why are you concerned about "where to put it"
> 
> The point is that I'm not. As integration between independent tools —
> taking your word at "provides primitives"


I'm not defending that viewpoint...


>  — it can only by solved by
> distributions providing both intending them to work together. So I leave
> it to packagers to solve appropriately in their distribution, and
> they'll choose a pkg.m4 appropriate for it.


If they only shipped u-config, then they cannot choose a pkg.m4
appropriate for that distribution at all.


> A pkg-config shouldn't be
> concerned with pkg.m4 any more than it is with pkg.el, pkg.py, pkg.lua,
> or other integrations.


pkg-config is a tool (or family of them) for informing build systems how
to find software build flags. It has good cause to be integrated with
build systems, and it *should* be concerned with this.

A pkg-config shouldn't be concerned with pkg.el, pkg.py, or pkg.lua
because none of those need to be running pkg-config in the first place.
They are programming languages, not build systems.

The comparison is fundamentally flawed.


> Historically Freedesktop.org pkg-config
> distributed a pkg.m4, which pkgconf copied (literally, and that's not a
> bad thing), and it's why it's still packaged that way. However, it's
> probably a mistake for distributions to reflect this in their package
> structure today.
> 
> Case in point: it's definitely a mistake for Debian and its derivatives.
> I cannot install pkg-config and pkgconf simultaneously *only* because
> they conflict on pkg.m4 despite it being the same script. If that file
> was provided by another package, such as through Automake or even its
> own package, then there would be no conflict. I could install both and
> choose the provider of the "pkg-config" command (and man page) through
> the alternatives system. In this case the bug report would go to Debian,
> not Automake.


The alternatives system is capable of handling the pkg.m4 file as well,
but Debian chose to mark pkgconf as "breaking" pkg-config to prevent
installing them both at the same time, because they intended to drop
pkg-config altogether.

I am not sure what you did to configure it via the alternatives system,
but I'm guessing you did this manually and outside of Debian's official
support, which means that it's not a Debian bug if you didn't tag pkg.m4
as being diverted -- it is a bug in your own testing.


-- 
Eli Schwartz

On 1/23/23 11:29 AM, Christopher Wellons wrote:
>> something like this and found `splitpath`
> 
> That was one of those functions needing revisiting, and I indeed rewrote
> it yesterday in terms of cut() in order to support --with-path.
> 
>> pkgconf pkg-config --variable pc_system_libdirs
> 
> I should have realized that since your patch set these! I didn't know
> you were copying a pkgconf-only feature. So unfortunately it won't work
> if the system has pkg-config (i.e. still the default for Debian and
> derivatives).
> 
> The Debian package (in debian/rules) uses dpkg-architecture alongside
> "gcc -print-multi-lib" to construct its system library path, which is
> hardcoded as a string in the binary with no way to access it directly.
> The include path is left to the default.
> 
> At least for testing — since it's been so useful to test with an
> identical configuration — I wonder if I can rig up "strings" and "grep"
> to grab it reliably enough if pc_system_libdirs fails (i.e. it's not
> pkgconf)…


pkgconf also has --dump-personality, which prints information in the
pkgconf-personality(5) manpage documented format, e.g.


SystemIncludePaths: /usr/include
SystemLibraryPaths: /usr/lib



-- 
Eli Schwartz

> pkgconf also has --dump-personality

I didn't know about this, thanks!

As it goes, I think I've got u-config just where I want it, so I cut a 
release, bring that over to w64devkit, build, test (mostly with .pc files 
I've written myself for GLFW, GLEW, etc.), and then notice something I 
missed. This time it's library order, particularly that the string set 
removing redundancies should be smarter. It's primarily an issue with 
static libraries, and as is often the case with Windows, import libraries, 
which behave similarly (and how I noticed this).

GCC+Binutils resolves symbols in one pass, left to right. It tracks 
undefined symbols and resolves them with definitions in later inputs. 
Inputs not referenced by an undefined symbol may be discarded. Supplying 
an input too early is like not including it at all. Thus definitions 
should follow, not precede, the inputs requiring them.

To make this more concrete, consider package A with libs "-lA -lC" (A 
needs definitions from C) and package B with libs "-lB -lC" (B needs 
definitions from C). Here's what happens:

pkg-config: -lA -lC -lB -lC
pkgconf: -lA -lB -lC
u-config: -lA -lC -lB

The ideal result is pkgconf, but pkg-config still works correctly. In some 
cases pkg-config produces the pkgconf output, but not reliably. u-config 
is the least correct. It nearly always works out because either the inputs 
are shared libraries or the main program references A which references C, 
keeping it "alive" for B. However, if the main program doesn't reference 
A, perhaps because it's an optional feature guarded by an #ifdef, then A 
is discarded, as is C, leaving B with undefined symbols. It's similar if A 
doesn't reference C strongly enough.

Note: There's probably some super-contrived case involving a circular 
dependency which technically requires the pkg-config output, but that's 
not worth considering.

Solution (I think): When removing redundant -l options, keep the last 
instance, not the first. It will need to buffer its full output (probably 
a linked list of arguments), then output arguments on a second pass. The 
string set should instead be a map that tracks the last position (e.g. an 
argument subscript) a particular option was seen. The second pass prints 
the last instance of each -l and the first instance of everything else.

This won't be difficult, but I'm going to think on it a bit longer in case 
I'm missing something. Or in case you point out something I missed!

Since I was aware that the library order matters for certain linkers,
investigating the -l output ordering was on my TODO list (right after I
solve the portage-autoconf mystery, which I haven't attended yet due to
getting side-tracked by some other things). But looks like you got to it
before me!

> pkg-config: -lA -lC -lB -lC
> pkgconf: -lA -lB -lC
> u-config: -lA -lC -lB

I agree that the pkgconf output is ideal in this case. But at the same
time, I think it's worth considering the "dumb" approach and not
filtering out duplicate `-l` at all. (I'm assuming this won't cause any
issues, but correct me if I'm wrong.)

This should be much simpler than the linked-list approach (I imagine
it'd just a single check). And if worst case scenario happens and the
circular dependency case turns out to be practical - this dumb approach
should handle that as well, whereas trying to solve the circular issue
while filtering duplicate would require much more heavyweight solution
I'd imagine.

- NRK

Small update: I've managed to figure out the portage issue. It was a
silly file-permission issue (portage didn't have permission for the log
file). With that fixed with a simple `chmod`, I can now proceed
forwards.

One bug that I noticed is that when --modversion is called, u-config
outputs the private library's version too.

Just from a quick glance at `(flags & ~Pkg_DIRECT) | flags`, it looked
like the OR operation was cancelling out the AND operation so I just
removed the OR and it seems to pass all tests.

I've attached the patch with the testcase. Though I'm not entirely sure
what the intention of that line was - and thus I'm not sure if the "fix"
is correct or not - I'll leave it up to you to verify :)

With the --modversion patch applied, I've went ahead and updated a
couple of packges on my system and it went smoothly so far. There's
still many more packages that need updating, I'll report back if I find
anything else.

- NRK

Patch accepted, thanks! Extra thanks for including a test.

> a silly file-permission issue

Great! I'm glad it was so simple.

> what the intention of that line was

The original line didn't make sense and was a mistake in d6d3e03. Your fix 
is what I had really intended.

> I'll report back if I find anything else

Thanks for all the testing! I currently consider u-config to be basically 
"done" and in maintenance mode. No new features or changes planned, just 
bug fixes and adding whatever options you find which are needed to support 
pkgconfg-based builds. Considering that it already supports a 25-year span 
of operating systems and compilers from multiple vendors, I don't expect 
it will ever require bit-rot maintenance.

One exception: Maybe at some point I'll write a proper man page.

Another update:

1. nodejs build requires `--silence-errors` (patch attached).
2. gtk+3 requires `--uninstalled`. So far I've ignored it.
3. gtk+3 requires `--atleast-version` but it's usage is very peculiar:
	pkg-config --atleast-version 2.57.2 glib-2.0 >= 2.57.2
   As far as I see, the `>=` should already have done the job.
4. gtk+3 and ffmpeg passes the "version" argument as a separate arg,
   which u-config doesn't seem to handle:
	pkg-config 'lib' '>=' '0.1.0'

I suspect many other real-world builds are doing 4. So the `process`
function probably needs to be redesigned a bit to make that work.

- NRK

> requires `--silence-errors`

Accepted. Also added to the --help listing since it's functional.

> requires `--uninstalled`

Accepted. No --help listing since it's not actually functional.

> which u-config doesn't seem to handle

Ha! I had deliberately added this "just in case" and even discussed it in 
my article as a pkg-config oddity, and something I took care to closely 
mimic in u-config. It's the whole reason I have the "Processor" type as a 
resumbable dependency traversal stack, and an "endprocessor" to validate 
that it's in a valid terminal state after all positional arguments are 
processed.

However, it seems I broke it sometime before I first checked it source 
control, so I can't trace it to a particular change. It probably happened 
the same time I split out "procfail" and ProcState.

Fortunately easy to fix (9231dda). Just needed to fix the persistence of 
the bottom of the stack between arguments, and skip the terminal check at 
the top-level since it's instead handled in endprocessor.

Another update (and most likely the final one) - I have now finished my
entire system update with u-config as the system pkg-config. For my
system at least, it seems that u-config is good enough.

However, a couple things:

1. Icecat (firefox) uses `--errors-to-stdout`. Low effort patch attached.
2. Aside from gtk+3, gnutls also uses `--atleast-version`. But unlike
gtk+3, the `--atleast-version` usage isn't redundant:
	pkg-config --atleast-version=0.23.11 p11-kit-1
(Though I'm not using p11-kit-1 so it didn't end up mattering for me).

And even before completing the full system update - commit `e00cb6e` was
something I had considered reverting in order to avoid false positives
since I'd much rather prefer the configuration check to fail instead of
marching forward and potentially causing issue down the line.

But then I realized that not all configuration script "fail" - some of
them try to be "smart" and do fallback stuff. So I kept the commit
during the update and instead kept a close eye on the log for any
`[^g-]-version` matches (which revealed the gnutls usage).

Overall it seems to me that the `--{atleast,exact,max}-version` might be
worth implementing - since real-world builds use it and both false
positive and false negative can end up having undesired outcome.

And on a side note - it seems that "bigger" projects tend to use more
niche stuff. And since I mainly try to use small/minimal software (one
of the main reason I'm using u-config) my system probably isn't as much
of a stress test compared to someone running kde/gnome or something
along those lines.

But in any case, I'll continue using u-config on my system as I'm happy
with the results :)

- NRK

> Low effort patch attached.

Forgot to attach it, done now :)

P.S: I had considered merging consecutive allocations into one (i,e
merging the out and err buffer) but that didn't seem like it's worth the
effort since the err buffer is so small.

- NRK

Wow, thanks for all the thorough testing!

I accepted your patch, then implemented --{atleast,exact,max}-version. The 
list of "deliberately omitted features" keeps shrinking! I had hoped to 
get by without all the redundant options, but if the options exist then 
people will use them regardless of how little sense it makes.

I was about to make a release yesterday, but then I remembered cgo can use 
pkg-confg and I hadn't tried it yet with u-config. So I did and discovered 
that I had missed "--" processing, which *is* an important feature. Then I 
wanted to wait for more feedback from you, which came right on time. With 
the "--" fix, I successfully wrote a small SDL2 program in Go on w64devkit 
yesterday, and it's neat that the "go build" now works there just as well 
as Linux.

> merging consecutive allocations into one

IMHO, a core benefit of region-based allocation is not worrying about 
these individual lifetimes and bits of waste. On average, non-recycled 
objects in arenas are probably no more wasteful than fragmentation in 
general purpose allocators. Though, just like you, I still can't help but 
notice and consider it anyway!

Of course, if there were many discarded objects with stack-incompatible 
lifetimes I'd consider using a freelist to recycle them.

While we're on the subject, in retrospect I probably still allocate too 
much on the stack, particularly in appmain. It doesn't matter much for 
u-config since nothing outlives appmain anyway. That is, it's not a game 
where the platform calls into the application every frame and it must 
persist state across these calls, or even across process instances. More 
conventionally there would be a high-level "push struct" for allocating 
structs, and objects like "Out" would always go in the arena, as would 
Processor, Pkgs, the global Env, and even Arenas — basically anything that 
currently has its address taken.