Come up with a strategy for writing automated tests

[alex]

No description provided.

[geofft]

Rust RFC 2318 is about experimental custom test frameworks. It was approved about a month ago and the tracking issue rust-lang/rust#50297 unsurprisingly doesn't show that anyone's started implementing it.

I wonder if it's worth us trying to implement it. We could just do something ourselves with using the kernel's own kallsyms API to find test functions and run them.

(The other thing I want here is tests against multiple different kernels and styles of kernel packaging, which is completely unrelated)

[alex]

Ok, here's my vision for how I'd like to be able to write tests.

First -- the type of tests I'm primarily interested in are testing, from userspace, the observable properties of the kernel module. So for a sysctl, this would be reading/writing /proc/sys/..., querying with sysctl(8), etc. I am not concerned (at this time) with either (1) compiling the kernel module source for userspace and writing unit tests against it, or (2) running in-memory tests against the loaded kernel module.

To that end, here's what I think a reasonable, and pracitcal, developer experience would be.

A tests/ directory at the top of the repo. Each subdirectory contains a kernel module as a crate (src/lib.rs, Cargo.toml, etc.), and a rust tests.rs written basically in current rust test framework style (#[test] etc.). Hopefully we won't need a Makefile for each, and we can find some way to share one between all of them. Example directory structure:

.
└── tests
    ├── printk
    │   ├── Cargo.toml
    │   ├── src
    │   │   └── lib.rs
    │   └── tests.rs
    └── sysctl
        ├── Cargo.toml
        ├── src
        │   └── lib.rs
        └── tests.rs

This defines two "test suites", sysctl and printk, each of which is a kernel module + a tests.rs test suite.

Execution would be cargo kernel-tests (or a better name), which by default runs all the test suites, but could also be run cargo kernel-tests printk (or perhaps cargo kernel-tests tests/printk/).

In terms of implementation, for each test suite:

1. The kernel module would be cross-compiled as usual (cargo xbuild ...) -- meaning this would require a host with linux kernel headers for now.
2. The tests.rs would be compiled as normal binary for a linux userspace target.
3. A qemu-system-x86_64 would be spawned (using /boot/initrd.img-$(uname -r) I guess?).
4. For each test function within the suite we'd insmod the module, run the test function, and then rmmod the module.

Obviously we can play with the details on which exact initrd we'd use, when to boot a new kernel vs. just reusing the existing session, whether to load/unload modules for each individual function.

One important thing to flag is we need to figure out how to get the compiled linux-kernel-module to be reused for each module, so we don't recompile the universe for each one.

An initial implementation could probably just be a shell script that loops over each test suite and compiles them and runs qemu + a library that all the tests run like:

#[test]
fn test_sysctl_bool_read() {
    with_kernel_module(|| {
        let data = fs::read_to_string("/proc/sys/rust/test/bool1");
        assert_eq!(data, b"0");
    });
}

WDYT?

[alex]

Via @sfackler, individual modules can be compiled as tests with rustc --test tests.rs. So I guess now it's really just a matter of writing a shell script to loop over the things.

@geofft does this sound reasonable to you? I don't know much about scripting qemu, is that something you're up on?

[alex]

I just thought of an optimization that'll get us to an MVP on testing faster: don't use QEMU this will run on the host.

This will obviously work fine on travis, and ATM I think we're both doing development in VMs, so it'll be usable (if not outstanding) for us as well.

Once we have working tests we can iterate for better isolation from there.

I'll take a swing at this once the sysctl patch is landed so I have something to test... or maybe I'll get impatient and start hacking on printk testing this afternoon before the 🏀 game, we'll see!