~skeeto/public-inbox

Re: Asynchronously Opening and Closing Files in Asyncio

Vladislav Yarmak
Details
Message ID
<20200907061601.60414aea@hp.lan>
DKIM signature
pass
Download raw message
Nice post, thanks!

I've been solving similar problem to decouple python daemon logging IO
from event loop in order to make logging completely non-blocking. I'm
pretty sure this particular case must be most popular file IO case for
async applications.

In my case it wasn't important to open log asynchronously, but it was
critical to have async writes.

It turns out, depths of Python stdlib hide
logging.handlers.QueueHandler and logging.handlers.QueueListener which
allow to split log message handler into separate thread from log
message emitters, making them communicate via synchronized queue.

Final solution is quite compact:
https://github.com/Snawoot/postfix-mta-sts-resolver/blob/589a59a90cf3f53148f9f3814ccd48cd92a9538f/postfix_mta_sts_resolver/utils.py#L26-L57

Usage example:
https://github.com/Snawoot/postfix-mta-sts-resolver/blob/589a59a90cf3f53148f9f3814ccd48cd92a9538f/postfix_mta_sts_resolver/daemon.py#L101-L106

It is convenient that you can still use logging as usual, without any
async obligations. You just have logging function which works in a
fire-and-forget fashion.

> You’d need an aprint() to complement print(), and so on, each
> returning an awaitable to be awaited.
> 
> This is one of the unfortunate downsides of async/await. I strongly
> prefer conventional, preemptive concurrency, but we don’t always have
> that luxury.

I was sceptical about Golang for a long time, but recently I started
practicing it. It's designed with concurrency in mind from very
beginning and goroutines have advantages of both cooperative and
preemptive multitasking. And also it doesn't have GIL, so your
goroutines will use all available CPUs without any additional effort
from your side.

From programmers point of view, coding with goroutines looks exactly
like coding with use of pthreads (even simpler, in fact). So, there are
no function "coloring", no need in sync/async library flavors and so
on. But these "threads" also have cost close to cost of python
coroutines. It sounds too good to be true, but I've rewritten bunch of
my Python/asyncio code in Go and I can say it's true. For this reason
(and many others I discovered after) I quit using Python and asyncio for
network applications in favor of Go.

-- 
Best Regards,
Vladislav Yarmak
Reply to thread Export thread (mbox)