Hi,

I, along with some others, have noticed my gemini capsule going down
regularly in the past two weeks. Unfortunately I don't yet have much
information about what's going on, but, after being told (indirectly)
that a couple other capsules have experienced what is likely the same
problem, it seems like a good idea to post what I do know here.

Since around 2021-03-26, my capsule gemini://calcuode.com/ has become
unresponsive a couple of times per day, requiring the server to be
restarted. At those times there are no errors shown, but looking
through the logs I can see systemd has restarted gmnisrv (sometimes
multiple times) after it failed due to a segmentation fault. I have not
been able to reproduce this failure. I can also see some SSL errors in
the logs, but I believe these are unrelated as the server continues to
work afterwards.

Possibly relevant info:
OS: Ubuntu 20.04 (which I am stuck with for now)
gmnisrv commit: 0dc0e4432a70eafde69509fde8a29802e46ae712
OpenSSL version: 1.1.1f (from package manager)

Others I know of who are experiencing what is likely the same problem:
kelbot of gemini://gemini.cyberbot.space/
gemini://sdf.org/


I know this isn't particularly helpful in terms of figuring out what's
going wrong or fixing it, but I'm really not sure where I'd start with
debugging and such. If there is anything I can do I would be very happy
to help.

Thanks,
Callum

I've experienced crashes as well, on both capsules I administrate.

The next step would be to debug further using a core dump. Since you're
on systemd, you can use `coredumpctl gdb`. The most useful information
will be the output of the command `bt full` - please send that to the
mailing list.

Ok. I've installed systemd-coredump and compiled gmnisrv with debugging
symbols, so hopefully I'll get something useful next time it crashes :)

Here it is:


coredumpctl info shows the following stack trace:

	#0  0x00007ff231f86005 __GI__IO_fread (libc.so.6 + 0x86005)
	#1  0x0000561b774b1611 client_writable (gmnisrv + 0x11611)
	#2  0x0000561b774b1fcd server_run (gmnisrv + 0x11fcd)
	#3  0x0000561b774a6621 main (gmnisrv + 0x6621)
	#4  0x00007ff231f270b3 __libc_start_main (libc.so.6 + 0x270b3)
	#5  0x0000561b774a42ee _start (gmnisrv + 0x42ee)



# coredumpctl gdb /usr/local/bin/gmnisrv

Core was generated by `/usr/local/bin/gmnisrv -C /etc/gmnisrv.ini'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007ff231f86005 in __GI__IO_fread (buf=0x7ff231aae060, size=1, count=4096, fp=0x0) at iofread.c:35
35	iofread.c: No such file or directory.

(gdb) bt full
#0  0x00007ff231f86005 in __GI__IO_fread (buf=0x7ff231aae060, size=1, count=4096, fp=0x0) at iofread.c:35
        _IO_acquire_lock_file = <optimized out>
        bytes_requested = 4096
        bytes_read = <optimized out>
#1  0x0000561b774b1611 in client_writable (server=0x7ffec27a5830, client=0x7ff231aae010)
    at src/server.c:428
        n = 0
        r = -1
        n = 51
        buf = "\027\003\003\000\347\343R\204\026\035\273\065\372\221-K\016\267\355\273g&\030t.\302\351YqQ\241\\\224j\tQc:L\326E\305\034g\340\314\375\f\313\321^\vQ\325\v\223\253(\230WGBQ 7\331|A)\240\236z:\241\213ꨣ\006U\031\233a<\033\311,\222a\203\345\b\224\225\353\323\363ƹ\034\026g^\210\307;\035s\311\352\vm\356%\256\062\371\367!\251,D\373\376\350G\304\345\207\003ӟ\317XZ\021\240v8PSw\266!\177sR\316\036B5\001\224m\242\345n[C\210\327\347\317\377\067\312<\031\261\316\066\252A\275 \357\210]\271v\004U\321\064M\241ք\316h\"\346m\"a\f\023\023\177\354\344Kk\273."...
        __PRETTY_FUNCTION__ = "client_writable"
#2  0x0000561b774b1fcd in server_run (server=0x7ffec27a5830) at src/server.c:602
        pi = 2
        s = CONNECTED
        i = 0
        act = {__sigaction_handler = {sa_handler = 0x561b774b1bd2 <handle_sigint>,
            sa_sigaction = 0x561b774b1bd2 <handle_sigint>}, sa_mask = {__val = {0 <repeats 16 times>}},
          sa_flags = 4, sa_restorer = 0x0}
        oint = {__sigaction_handler = {sa_handler = 0x0, sa_sigaction = 0x0}, sa_mask = {__val = {0,
              94675992377064, 140732161218144, 8, 140678196922479, 140678201874816, 8, 94675992386976,
              140678197223437, 94675992377024, 10, 140678198642112, 94675965534800, 3432,
              140678197223437, 10}}, sa_flags = 0, sa_restorer = 0x0}
        oterm = {__sigaction_handler = {sa_handler = 0x0, sa_sigaction = 0x0}, sa_mask = {__val = {0,
              94675992377064, 140732161218144, 8, 140678196922479, 140678201874816, 8, 94675992386976,
              140678197223437, 94675992377024, 10, 140678198642112, 94675965534800, 3432,
              140678197223437, 10}}, sa_flags = 0, sa_restorer = 0x0}
        r = 2
        __PRETTY_FUNCTION__ = "server_run"
#3  0x0000561b774a6621 in main (argc=3, argv=0x7ffec27a5978) at src/main.c:60
        conf = {tls = {store = 0x561b78e2f230 "/var/lib/gemini/certs", organization = 0x0,
            ssl_ctx = 0x561b78e2f7b0}, hosts = 0x561b78e2f250, binds = 0x561b78e2f1e0}
        confpath = 0x7ffec27a6eea "/etc/gmnisrv.ini"
        c = -1
        r = 0
        server = {conf = 0x7ffec27a5800, fds = 0x561b78e563d0, nfds = 4, fdsz = 1024, nlisten = 2,
          clients = 0x7ff231aae010, nclients = 2, clientsz = 1024, run = true}

I still haven't worked out what precisely is going wrong, but I have
managed to reliably reproduce the issue by sending a ton of requests
at once. Basically and extremely easy DoS attack.

#!/bin/sh
while true
do
   gmni -j once gemini://xxxxx/ &
done

Thanks for the consistent reproduction! That helped a ton. This should
be fixed as of b9a92193e96bbe621ebc8430d8308d45a5b86cef.