The future of Offpunk: UNIX command-line heaven and packaging hell

Message ID
DKIM signature
Download raw message
by Ploum on 2023-10-01


> A story about how the UNIX philosophy made me develop tools I’m 
actually proud of and why packaging is holding me back.

Two years ago, I decided that I wanted to be able to browse Gemini while 
offline. I started to add a permanent cache to Solderpunk’s AV-98, the 
simplest and first Gemini browser ever. It went surprisingly well. Then, 
as the excellent forlater.email service went down for a week, I thought 
that I would add a quick and hackish HTTP support to it. Just a 
temporary experiment.

The same week, I serendipitously stumbled upon chafa, an image rendering 
tool which was on my computer because of neofetch. I thought it would be 
funny to have pictures rendered in webpages in my terminal. Just an 
experiment to take some funny screenshots, nothing more.

But something really surprising happened: it was working. It was really 
useful. I was really using it and, after adding support for RSS, I 
realised that this experiment was actually working better for me than 
forlater.email and newsboat. Offpunk was born without really thinking 
about it and became a real project with its own philosophy.

Offpunk, a command line to browse Web, Gemini and Gopher while offline

Born on Gemini, I wanted Offpunk to keep its minimalistic roots: keeping 
dependencies under control (making them optional and implementing the 
underlying feature myself as soon as it makes sense), keeping it simple 
(one single runnable python script), caring as much as possible about 
older versions of python, listening to people using it on very minimal 
systems. I also consciously choose to use only solutions that have been 
time-trial-tested. I’ve spent too many years of my life falling for the 
"new-trendy-technology" and learned from those mistakes. The one-file 
aspect assured that it was really easy to use and to hack on it: open 
the file, modify something, run it.

I’m not a good developer. Anything more complex than that is too much 
for my taste. Unless forced, I’ve never used an IDE, never understood 
complex toolchains nor packaging. I modify files with (neo)vim (without 
any plugin), compile from the command line and run the resulting binary 
(not even needing the second step with python). Life is too short for 
making it more complex than that. I like to play with the code, not to 
learn tools that would do it for me.

But offpunk.py was becoming a bit fat. 4500 lines of organic python 
which have grown over an AV-98 structured to be a test bed for an 
experimental protocol. The number of people able to understand its code 
entanglement was varying between 0 and 1, depending on the quality of my 
morning Earl Grey.

I wanted to make life easier for contributors. I also realised that some 
features I developed might be useful without offpunk. So I stepped into 
a huge refactorisation and managed to split offpunk into several 
components. My goal was to separate the code into multiple individual 
components doing one thing and doing it well. And, to my own surprise, I 


I called the first component "netcache". Think of netcache as a cached 
version of wget. If possible, netcache will give you a cached version of 
the URL you are asking. If no cache or too old and if allowed to go 
online, netcache will download it.

It means that if you like Offpunk’s core concept but don’t like the 
interface and want, for example, a GUI, you could write your own browser 
that would, using netcache, share the cache with Offpunk.

Netcache is currently working just well enough for my needs but could be 
a lot better. I should, for example, investigate replacing the network 
code by libcurl and implementing support for multithreaded concurrent 


Coloured output in your terminal is done through a standard called ANSI. 
As I wrote the first HTML to ANSI renderer for offpunk, I started to 
understand how awful the HTML standard was. Armed with that experience, 
I started a second renderer and, to be honest, it is actually not that 
bad. I’m even proud of it.

Ansicat is really useful when in a terminal because it will render HTML 
and gemtext in a good, readable way. If the optional library python- 
readability is present, ansicat will try to extract the main content 
from a web page (and, yes, python-readability is one dependency I would 
like to reimplement someday).

With netcache and ansicat, you can thus already do something like:

netcache https://ploum.net | ansicat --format=html

Yes, it works. And yes, as a UNIX junkie, I was completely excited the 
first time it worked. Look mum, I’m Ken Thompson! Making ansicat a 
separate tool made me think about adding support for other formats. Like 
PDF or office documents. How cool would it be to have a singe cat 
command for so many different formats?


While netcache and ansicat were clear components I wanted to split from 
Offpunk’s core since the start of the refactoring, another tool appeared 
spontaneously: opnk.

Opnk (Open-like-a-punk) is basically a wrapper that will run ansicat on 
any file given. If given a URL, it will ask netcache for the file. 
Result will be displayed in less (after passing through ansicat, of 

If ansicat cannot open the file, opnk fallbacks on xdg-open.

That looks like nothing but it proved to be massively useful in my 
workflow. I already use opnk every day. Each time I want to open a file, 
I don’t think about the command, I type "opnk". It even replaced cat for 
many use cases. I’m even considering renaming it "opn" to save one 
character. Using opnk also explains why I want to work on supporting 
PDF/office documents with ansicat. That would be one less opportunity to 
leave the terminal.


Through this architecture, Offpunk became basically an interface above 
opnk. And this proved to work well. Many longstanding bugs were fixed, 
performance and usability were vastly improved.

Everything went so well that I dreamed releasing offpunk 2.0, netcache, 
ansicat and opnk while running naked with talking animals in a flower 
field under a rainbow. Was it really Earl Grey in the cup that day?

Packaging Offpunk.py

Now for the bad news.

As expected, the refactoring forced me to break my "one-single-python- 
file" rule.

I felt guilty for those people who told me about using offpunk on very 
minimal systems, sometimes from a USB key. But I thought that this was 
not a real problem. Instead of one python script, I had four of them 
(and a fifth file containing some shared code). That should not be that 
much of a problem, isn’t it?

Well, python packaging systems would like to disagree. Flowers fade, the 
rainbow disappears behind black and heavy clouds while animals start to 
look at me with a devilish look and surprisingly sharp teeth.

I’ve spent many hours, asked several people on the best way to package 
multiple python files without making the whole thing a module. Without 
success. Hopefully, the community is really helpful. David Zaslavsky 
stepped on the mailing list to give lots of advice and, as I was 
discouraged, Austreelis started to work really hard to make offpunk both 
usable directly and packagable. I’m really grateful for their help and 
their work. But, so far, without clear success. I feel sad about the 
amount of energy required to address something as simple as "I’ve 5 
python files which depend on each other and I want to be able to launch 
them separately".

Austreelis’s branch where she works on making offpunk "packageable".

The software is working really well. The refactoring allowed me to fix 
longstanding bugs and to improve a lot of areas while adding new 
features (colour themes anyone?) On my computer, I added four aliases in 
my zsh config: offpunk, opnk, ansicat and netcache. Each alias runs the 
corresponding python file. Nothing fancy and I want to keep it that way. 
I know for a fact that several users are doing something similar: git 
clone then run it from an arbitrary location.

Keeping things as simple as that is the main philosophical goal behind 
offpunk. It’s an essential part of the project. If people want to use 
pip or any other tool to mess up their computer configuration, that’s 
their choice. But it should never be required.

Which means that I’m now in a very frustrating position: Offpunk 2.0 is 
more than ready from a code point of view. But it cannot be shipped 
because there’s currently no easy way to package it. The pyproject.toml 
file had become an obstacle to the whole development process.

I’m contemplating putting everything back in one big file. Or removing 
the pyprojects.toml file from the repository and releasing offpunk "as 
it is".

Some will call me an old conservative fart for refusing to use one of 
those gazillion shiny packaging system. Others will judge me as a pretty 
poor programmer if I managed to do 20 years of Python without ever 
understanding pip nor using an IDE.

They are probably right. What would you seriously expect from someone 
doing a command-line tool to browse Gemini and Gopher?

But there’s maybe an easier solution than to change my mind and 
offpunk’s core philosophy. A simple solution that I missed. If that’s 
the case, don’t hesitate to drop a word on the devel mailing-list, 
Austreelis and I will be happy to hear about your opinion and your 

offpunk-devel mailing list

While you are at it, bug reports and feedback are also welcome. I’ve 
this odd custom of finding embarrassing bugs only hours after a release. 
I really hope to do better with offpunk 2.0.

And after we’ve solved that little packaging anecdote together, I will 
happily return to my bare neovim to code all the ideas I want to 
implement for 2.1, 2.2 and many more releases to come.

To unsubscribe, send an email to ~lioploum/en+unsubscribe@lists.sr.ht
Reply to thread Export thread (mbox)