Invalid memory address or nil pointer dereference within DQLite library about go-dqlite HOT 10 CLOSED

Can you also check if you have a libraft or similar package installed and what version?

from go-dqlite.

Certainly, I have

➜  ~ pacman -Q | grep raft
raft 0.11.3-1

Also possibly useful:

➜  ~ cat /usr/lib/pkgconfig/raft.pc
prefix=/usr
exec_prefix=${prefix}
libdir=${exec_prefix}/lib
includedir=${prefix}/include

Name: raft
Description: C implementation of the Raft Consensus protocol
Version:  0.11.2
Libs: -L${libdir} -lraft
Cflags: -I${includedir}

➜  ~ pacman -Ql raft 
raft /usr/
raft /usr/include/
raft /usr/include/raft.h
raft /usr/include/raft/
raft /usr/include/raft/fixture.h
raft /usr/include/raft/uv.h
raft /usr/lib/
raft /usr/lib/libraft.so
raft /usr/lib/libraft.so.0
raft /usr/lib/libraft.so.0.0.7
raft /usr/lib/pkgconfig/
raft /usr/lib/pkgconfig/raft.pc
raft /usr/share/
raft /usr/share/licenses/
raft /usr/share/licenses/raft/
raft /usr/share/licenses/raft/LICENSE

from go-dqlite.

what's the output of cat /lib/modules/$(uname -r)/build/.config | grep AIO ?

from go-dqlite.

It is

➜  ~ cat /lib/modules/$(uname -r)/build/.config | grep AIO
CONFIG_AIO=y
CONFIG_DELL_WMI_AIO=m

from go-dqlite.

As this bug is being a particulate annoyance to me, I am willing to help in any way. If you would like I can recompile this library with any debug code injected here at the exception place.

Except I don't know go so you would have to exactly tell me what to print and how

from go-dqlite.

It'd likely help a fair bit if you could rebuild libraft and libdqlite to use the current master version, that would remove some potential variables and help with debugging.

If you have the debug symbols (which will be the case with a rebuild), then you should be able to run gdb against LXD and get a backtrace.

from go-dqlite.

Looks like node is NULL here https://github.com/canonical/dqlite/blob/22edfc5453d9046e6cfedbcb1b67af3a7a85a7f7/src/server.c#L662

The node output parameter of https://github.com/canonical/dqlite/blob/22edfc5453d9046e6cfedbcb1b67af3a7a85a7f7/src/server.c#L149 is probably NULL because this https://github.com/canonical/dqlite/blob/22edfc5453d9046e6cfedbcb1b67af3a7a85a7f7/src/server.c#L22 fails.

You can try setting the environment variables LIBRAFT_TRACE=1 and LIBDQLITE_TRACE=1 before starting lxd but I doubt they will give much extra information in this case.

You could already try and recompile dqlite and spread some printf statements in dqlite__init to see which call actually fails, if it's a raft call you will probably need to do the same printf debugging after recompiling https://github.com/canonical/raft

I will try to look at this beginning of the week, this PR canonical/dqlite#351 should already eliminate the segfault.

from go-dqlite.

I have made some debugging sessions, see them here (in retrospect, you can completely ignore these, as I already found the issue) :

https://asciinema.org/a/5TkKsFWPmqyOt1UNefjCkH9YE
https://asciinema.org/a/x08wglxLX5ctPvAXbMGELHCEg

And most importantly:

https://asciinema.org/a/ngZtefehDcYQcUx9z41eZNrQR

The error seems to be related to a failed filesystem check, specifically call probeDirectIO returns 18 ( which would make sense, as my root as mentioned earlier is... hackily built together aufs)

from go-dqlite.

Even more specifically, call rv = fcntl(fd, F_SETFL, flags | UV_FS_O_DIRECT); in UvOsSetDirectIo returns -1

from go-dqlite.

I mitigated the problem by creating a logical volume and mounted it over /var/lib/lxd

As far as I am immediately concerned this problem is mitigated however something like "Unsupported filesystem" message would be nice

from go-dqlite.