drgn doesn't handle XZ compressed kernel modules

[danobi]

Built and installed from source on fedora 30:

$ cd /usr/lib/modules/5.1.18-300.fc30.x86_64/kernel 
$ ls
arch  crypto  drivers  fs  kernel  lib  mm  net  security  sound  virt
$ rpm -qf arch
kernel-core-5.1.18-300.fc30.x86_64
$ sudo drgn
could not get debugging information for:
kernel (could not find vmlinux)
/usr/lib/modules/5.1.18-300.fc30.x86_64/kernel/drivers/net/usb/ipheth.ko.xz (could not get section addresses: libelf error: invalid `Elf' handle)
/usr/lib/modules/5.1.18-300.fc30.x86_64/kernel/crypto/ccm.ko.xz (could not get section addresses: libelf error: invalid `Elf' handle)
/usr/lib/modules/5.1.18-300.fc30.x86_64/kernel/drivers/input/misc/uinput.ko.xz (could not get section addresses: libelf error: invalid `Elf' handle)
/usr/lib/modules/5.1.18-300.fc30.x86_64/kernel/fs/fuse/fuse.ko.xz (could not get section addresses: libelf error: invalid `Elf' handle)
... 167 more
drgn 0.0.1 (using Python 3.7.4)
For help, type help(drgn).
>>> import drgn
>>> from drgn import cast, container_of, execscript, NULL, Object, reinterpret
>>> from drgn.helpers.linux import *
>>>
>>> find_task(prog, 1).cred.user.locked_vm
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/usr/local/lib64/python3.7/site-packages/drgn-0.0.1-py3.7-linux-x86_64.egg/drgn/helpers/linux/pid.py", line 117, in find_task
    return pid_task(find_pid(prog_or_ns, pid), prog['PIDTYPE_PID'])
  File "/usr/local/lib64/python3.7/site-packages/drgn-0.0.1-py3.7-linux-x86_64.egg/drgn/helpers/linux/pid.py", line 35, in find_pid
    ns = prog_or_ns['init_pid_ns'].address_of_()
KeyError: 'init_pid_ns'

[danobi]

So it looks like the issue is fedora likes to xz compress kernel modules. Can drgn have support to decompress?

[danobi]

Would help with debugging bpftrace/bpftrace#881 on my laptop

[osandov]

I have a change in the works which will incidentally add support for compressed modules (that error message sucks, I'm fixing that, too). But, on Fedora, the compressed modules don't have debugging information anyways. You need to install the debuginfo package instead. Something like:

sudo dnf install --enablerepo=updates-debuginfo kernel-debuginfo

[danobi]

Ah ok. I'll give it a try when I'm on a stronger internet connection. Feel free to rename/close the issue.

[danobi]

Looks like drgn is having trouble locating the debug files. Not sure if you use debuglink but it seems like fedora has put the debuginfos in a non-standard location.

$ pwd
/usr/lib/modules/5.2.11-200.fc30.x86_64/kernel/kernel/trace

$ readelf -p .gnu_debuglink ring_buffer_benchmark.ko

String dump of section '.gnu_debuglink':
  [     0]  ring_buffer_benchmark.ko.debug
  [    23]  j

$  ll /usr/lib/debug/lib/modules/5.2.14-200.fc30.x86_64/kernel/kernel/trace/ring_buffer_benchmark.ko.debug
-r--r--r--. 1 root root 229K Sep 10 05:34 /usr/lib/debug/lib/modules/5.2.14-200.fc30.x86_64/kernel/kernel/trace/ring_buffer_benchmark.ko.debug

[osandov]

Hmm, drgn knows to look in that directory for kernel module debug info. Can you share the output of uname -r and the output you get from drgn?

[danobi]

$ uname -r
5.2.11-200.fc30.x86_64
$ sudo drgn
[sudo] password for dlxu:
could not get debugging information for:
kernel (could not find vmlinux)
/usr/lib/modules/5.2.11-200.fc30.x86_64/kernel/drivers/char/lp.ko.xz (could not get section addresses: libelf error: invalid `Elf' handle)
/usr/lib/modules/5.2.11-200.fc30.x86_64/kernel/drivers/parport/parport.ko.xz (could not get section addresses: libelf error: invalid `Elf' handle)
/usr/lib/modules/5.2.11-200.fc30.x86_64/kernel/drivers/net/usb/ipheth.ko.xz (could not get section addresses: libelf error: invalid `Elf' handle)
/usr/lib/modules/5.2.11-200.fc30.x86_64/kernel/drivers/thunderbolt/thunderbolt.ko.xz (could not get section addresses: libelf error: invalid `Elf' handle)
... 162 more
drgn 0.0.1 (using Python 3.7.4)
For help, type help(drgn).
>>> import drgn
>>> from drgn import cast, container_of, execscript, NULL, Object, reinterpret
>>> from drgn.helpers.linux import *
>>> find_task(prog, 1).cred.user.locked_vm.counter
Traceback (most recent call last):
  File "/usr/lib64/python3.7/code.py", line 90, in runcode
    exec(code, self.locals)
  File "<console>", line 1, in <module>
  File "/usr/local/lib64/python3.7/site-packages/drgn-0.0.1-py3.7-linux-x86_64.egg/drgn/helpers/linux/pid.py", line 117, in find_task
    return pid_task(find_pid(prog_or_ns, pid), prog['PIDTYPE_PID'])
  File "/usr/local/lib64/python3.7/site-packages/drgn-0.0.1-py3.7-linux-x86_64.egg/drgn/helpers/linux/pid.py", line 35, in find_pid
    ns = prog_or_ns['init_pid_ns'].address_of_()
KeyError: 'init_pid_ns'

[osandov]

Ah okay you're running 5.2.11-200.fc30.x86_64, but the dnf command installed the debugging information for the latest kernel, 5.2.14-200.fc30.x86_64. You can either reboot into that kernel or try:

sudo dnf install --enablerepo=updates-debuginfo kernel-debuginfo-$(uname -r)

[danobi]

Ran

sudo dnf install --enablerepo=updates-debuginfo kernel-debuginfo-$(uname -r)

again and it works now. I think my kernel version and debuginfo were just constantly skewed.

[jfernandez]

@osandov did you add support for compressed modules?

[osandov]

@jfernandez gz and xz compressed modules should work as long as drgn was compiled with elfutils version >= 0.175. I just realized that zstd compressed modules need a code change in drgn, so let me know if that affects you.

[jfernandez]

@osandov I realized that my config was using zstd compression. I'm using Arch Linux drgn package. I disabled compression to get around this.