Hi Drew,
you're heavily complaining (quite rightfully so) about the state of
packaging in Python, but I don't see any mention or link to what exactly
are "the needs of any of the distros, despite our repeated pleas." -
could you please elaborate?
- Anselm
On Thu Nov 18, 2021 at 11:44 AM CET, Anselm Kiefner wrote:
> you're heavily complaining (quite rightfully so) about the state of> packaging in Python, but I don't see any mention or link to what exactly> are "the needs of any of the distros, despite our repeated pleas." -> could you please elaborate?
Here are a few common use-cases which are forgotten or explicitly
ignored by various newfangled Python build systems/package managers/bit
twiddlers:
- Building & running tests without the network and without fetching
dependencies during the build
- Installing into an arbitrary prefix
- Installing into an arbitrary prefix within a DESTDIR
- Working with sysroots when linking to native code
- Correct use of pkg-config
I don't understand the requirement to build and run tests without
fetching dependencies during the build. It seems plainly absurd to me,
as you have to install requirements to run the tests.
I guess installing into arbitrary prefixes is something you need to
build the rpm/deb packages, that it somehow detects what was installed
and creates the distro package from that?
I've no idea what the problem is with sysroots.
Isn't pkg-config the responsibility of the tool that creates the distro package?
I think there is a lot of confusion here about what is the
responsibility of the distro and their maintainers, and what is the
responsibility of the build tool, and that may be the source of the
problem? As a package maintainer in Python, I have sometimes attempted
to make packages for Linux distro's and to be honest I've always
failed to find any useful information on how to do it and given up
every time. I suspect that makes it entirely impossible for Python
people to know what distros want from them.
Of course, the absurd proliferation of distros aren't helping, but
that's a different topic.
On Fri, Nov 19, 2021 at 11:39 AM Drew DeVault <sir@cmpwn.com> wrote:
>> On Thu Nov 18, 2021 at 11:44 AM CET, Anselm Kiefner wrote:> > you're heavily complaining (quite rightfully so) about the state of> > packaging in Python, but I don't see any mention or link to what exactly> > are "the needs of any of the distros, despite our repeated pleas." -> > could you please elaborate?>> Here are a few common use-cases which are forgotten or explicitly> ignored by various newfangled Python build systems/package managers/bit> twiddlers:>> - Building & running tests without the network and without fetching> dependencies during the build> - Installing into an arbitrary prefix> - Installing into an arbitrary prefix within a DESTDIR> - Working with sysroots when linking to native code> - Correct use of pkg-config
On Fri Nov 19, 2021 at 11:47 AM CET, Lennart Regebro wrote:
> I don't understand the requirement to build and run tests without> fetching dependencies during the build. It seems plainly absurd to me,> as you have to install requirements to run the tests.
The distro will install them through its packages, not via PyPI.
The rest of your comments similarly misunderstand the situation.
> I think there is a lot of confusion here about what is the> responsibility of the distro and their maintainers, and what is the> responsibility of the build tool, and that may be the source of the> problem? As a package maintainer in Python, I have sometimes attempted> to make packages for Linux distro's and to be honest I've always> failed to find any useful information on how to do it and given up> every time. I suspect that makes it entirely impossible for Python> people to know what distros want from them.
Don't make packages for Linux distro's, it's a fool's errand. Just
publish to PyPI and let the distros make their own packages if they need
your module.
On Fri, Nov 19, 2021 at 11:53 AM Drew DeVault <sir@cmpwn.com> wrote:
>> On Fri Nov 19, 2021 at 11:47 AM CET, Lennart Regebro wrote:> > I don't understand the requirement to build and run tests without> > fetching dependencies during the build. It seems plainly absurd to me,> > as you have to install requirements to run the tests.>> The distro will install them through its packages, not via PyPI.
I understand that. This does not in any way answer my questions or comments.
> The rest of your comments similarly misunderstand the situation.
No, they don't.
> Don't make packages for Linux distro's, it's a fool's errand. Just> publish to PyPI and let the distros make their own packages if they need> your module.
Then what is the problem?
Hey Lennart,
On Fri, Nov 19, 2021 at 11:57:24AM +0100, Lennart Regebro wrote:
>On Fri, Nov 19, 2021 at 11:53 AM Drew DeVault <sir@cmpwn.com> wrote:>>>> On Fri Nov 19, 2021 at 11:47 AM CET, Lennart Regebro wrote:>> > I don't understand the requirement to build and run tests without>> > fetching dependencies during the build. It seems plainly absurd to me,>> > as you have to install requirements to run the tests.>>>> The distro will install them through its packages, not via PyPI.>>I understand that. This does not in any way answer my questions or comments.>>> The rest of your comments similarly misunderstand the situation.>>No, they don't.>>> Don't make packages for Linux distro's, it's a fool's errand. Just>> publish to PyPI and let the distros make their own packages if they need>> your module.>>Then what is the problem?
As someone who struggles to package python projects on NixOS, the main
problems I run into are packages that try to download dependencies at
build time. NixOS doesn't allow such things and it interferes with the
hermetic nature of NixOS.
The problem is the Python dev culture that doesn't seem to care or take
into consideration packaging requirements so that end users can
eventually install their software.
Poetry has made things a bit nicer, but I still run into issues with
packages that fail to build because some transitive dependency tries to
do something dumb inside setup.py. Python is by far one of the worse
offender out of all of the languages I've tried packaging.
In general, for Python devs who want to make it easier on package maintainers:
- Don't fetch your dependencies at build time, or if you do, provide
an environment variable that points to the result of the download.
- Don't try to get fancy inside setup.py, the simpler and more
"typical" a package is, the easier it is to package.
- Try not to depend on so many packages, they will probably end up
doing something dumb that will make your project more difficult to
package.
Overall I've found that most problems are caused by packages that try to
do something stupid, exacerbated by the fact that pip doesn't try to
stop or discourage devs from doing stupid things.
because pip is such clusterfuck, I'm not sure if Python developers have
any hope here without a culture shift, a bit of education, and perhaps
adoption of a more sane packaging solution like poetry.
Cheers,
Will
--
https://jb55.com
William Casarin:
> As someone who struggles to package python projects on NixOS, the main> problems I run into are packages that try to download dependencies at> build time. NixOS doesn't allow such things and it interferes with the> hermetic nature of NixOS.
I'm not entirely sure what you mean with "build time" here, as opposed
to something else?
But since Python has to have a packaging system that doesn't rely on
the operating system
(as established elsewhere in this discussion) these packages has to be
downloaded.
Unless the Python package in question has C extensions or similar,
there is no other "build time" at all.
> The problem is the Python dev culture that doesn't seem to care or take> into consideration packaging requirements so that end users can> eventually install their software.
I am here right now trying to care, and I get zero useful answers or
explanations,
> - Don't fetch your dependencies at build time, or if you do, provide> an environment variable that points to the result of the download.
I don't think any package "downloads" requirements. When you install a
Python package,
the package dependency system will install the dependencies. Yes, they
will fetch them from the internet.
How else should they do that?
> - Don't try to get fancy inside setup.py, the simpler and more> "typical" a package is, the easier it is to package.
This is a well known issue, and has been for years, and in fact, the
systems Drew complained
about are to a large extent designed exactly to solve the problems
that fancy setup.py's created.
So that's a non-issue. In fact, setup.py is deprecated and on the way
out since many years.
> - Try not to depend on so many packages, they will probably end up> doing something dumb that will make your project more difficult to> package.
Yeah, you won't get anywhere with that. People don't want to reinvent
the wheel. One of the things with
Python is that there are packages that do what you want already. Your
choice is to use that package,
or rewrite it, with new bugs.
> Overall I've found that most problems are caused by packages that try to> do something stupid
Sure, but that's not what Drew's blog post was about.
> because pip is such clusterfuck, I'm not sure if Python developers have> any hope here without a culture shift, a bit of education, and perhaps> adoption of a more sane packaging solution like poetry.
A solution that Drew called a problem.