Integration testing using virtual machines#
This tutorial aims to be compatible with NixOS release 22.11.
What will you learn?#
This tutorial introduces the functionality of Nixpkgs to write automated tests to debug NixOS configurations independent of a working NixOS installation.
What do you need?#
A working installation of Nix Package Manager or NixOS.
Basic knowledge of the Nix language.
Basic knowledge of NixOS configuration.
Introduction#
Nixpkgs provides a test environment to automate integration testing for distributed systems. It allows defining tests based on a set of declarative NixOS configurations and using a Python shell to interact with them through QEMU as the backend. Those tests are widely used to ensure that NixOS works as intended, so in general they are called NixOS Tests. They can be written and launched outside of NixOS, on any Linux machine[1].
The nixosTest
function#
NixOS VM tests are defined using the nixosTest
function.
The pattern for NixOS VM tests looks like this:
1let
2 nixpkgs = fetchTarball "https://github.com/NixOS/nixpkgs/tarball/nixos-22.11";
3 pkgs = import nixpkgs { config = {}; overlays = []; };
4in
5 pkgs.nixosTest {
6 name = "test-name";
7 nodes = {
8 machine1 = { config, pkgs, ... }: {
9 # ...
10 };
11 machine2 = { config, pkgs, ... }: {
12 # ...
13 };
14 }
15 testScript = { nodes, ... }: ''
16 # ...
17 '';
18 }
The function nixosTest
takes an attribute set that follows the module convention to specify the test.
Because the attribute set only defines options, one can use the abbreviated form of the module convention.
The attribute set needs to define the following options:
name
defines the name of the test.nodes
contains a set of named configurations, because a test script can involve more than one virtual machine. Each virtual machine is setup using a NixOS configuration.testScript
defines the Python test script, either as literal string or as a function that takes anodes
attribute. This Python test script can access the virtual machines via the names used for thenodes
. It has super user rights in the virtual machines. In the Python script is each virtual machine is accessible via themachine
object. NixOS provides various methods to run tests on these configurations.
The test framework automatically starts the virtual machines and runs the Python script.
Minimal example#
As a minimal test on the default configuration, we will check if the user root
and alice
can run Firefox.
We will build the example up from scratch.
As recommended we use an explicitly pinned version of Nixpkgs, and explicitly set configuration options and overlays to avoid them being inadvertently overridden by global configuration:
1let
2 nixpkgs = fetchTarball "https://github.com/NixOS/nixpkgs/tarball/nixos-22.11";
3 pkgs = import nixpkgs { config = {}; overlays = []; };
4in
5 pkgs.nixosTest {
6 # ...
7 }
Options#
Name#
Label the test with a descriptive name such as “minimal-test”:
1name = "minimal-test";
Nodes#
Because this example only uses one virtual machine, the node we specify is simply called machine
.
This name is arbitrary and can be chosen freely.
As configuration you use the relevant parts of the default configuration, that we used in a previous tutorial:
1nodes.machine = { config, pkgs, ... }: {
2 users.users.alice = {
3 isNormalUser = true;
4 extraGroups = [ "wheel" ];
5 packages = with pkgs; [
6 firefox
7 tree
8 ];
9 };
10
11 system.stateVersion = "22.11";
12};
Test script#
This is the test script:
1machine.wait_for_unit("default.target")
2machine.succeed("su -- alice -c 'which firefox'")
3machine.fail("su -- root -c 'which firefox'")
This Python script is referring to machine
which is the name chosen for the virtual machine configuration used in the nodes
attribute set.
The script waits until systemd reaches default.target
.
It uses the su
command to switch between users and the which
command to check if the user has access to firefox
.
It expects that the command which firefox
to succeed for user alice
and to fail for root
.
This script will be the value of the testScript
attribute.
Test file#
The complete minimal-test.nix
file content looks like the following:
let
nixpkgs = fetchTarball "https://github.com/NixOS/nixpkgs/tarball/nixos-22.11";
pkgs = import nixpkgs { config = {}; overlays = []; };
in
pkgs.nixosTest {
name = "minimal-test";
nodes.machine = { config, pkgs, ... }: {
users.users.alice = {
isNormalUser = true;
extraGroups = [ "wheel" ];
packages = with pkgs; [
firefox
tree
];
};
system.stateVersion = "22.11";
};
testScript = ''
machine.wait_for_unit("default.target")
machine.succeed("su -- alice -c 'which firefox'")
machine.fail("su -- root -c 'which firefox'")
'';
}
Running tests#
To set up all machines and run the test script:
$ nix-build minimal-test.nix
...
test script finished in 10.96s
cleaning up
killing machine (pid 10)
(0.00 seconds)
/nix/store/bx7z3imvxxpwkkza10vb23czhw7873w2-vm-test-run-minimal-test
Interactive Python shell in the virtual machine#
When developing tests or when something breaks, it’s useful to interactively tinker with the test or access a terminal for a machine.
To start an interactive Python session with the testing framework:
$ $(nix-build -A driverInteractive minimal-test.nix)/bin/nixos-test-driver
Here you can run any of the testing operations.
Execute the testScript
attribute from minimal-test.nix
with the test_script()
function.
If a virtual machine is not yet started, the test environment takes care of it on the first call of a method on a machine
object.
But you can also manually trigger the start of the virtual machine with:
>>> machine.start()
for a specific node,
or
>>> start_all()
for all nodes.
You can enter a interactive shell on the virtual machine using:
>>> machine.shell_interact()
and run shell commands like:
uname -a
Linux server 5.10.37 #1-NixOS SMP Fri May 14 07:50:46 UTC 2021 x86_64 GNU/Linux
Re-running successful tests
Because test results are kept in the Nix store, a successful test is cached. This means that Nix will not run the test a second time as long as the test setup (node configuration and test script) stays semantically the same. Therefore, to run a test again, one needs to remove the result.
If you would try to delete the result using the symbolic link, you will get the following error:
nix-store --delete ./result
finding garbage collector roots...
0 store paths deleted, 0.00 MiB freed
error: Cannot delete path '/nix/store/4klj06bsilkqkn6h2sia8dcsi72wbcfl-vm-test-run-unnamed' since it is still alive. To find out why, use: nix-store --query --roots
Instead, remove the symbolic link and only then remove the cached result:
rm ./result
nix-store --delete /nix/store/4klj06bsilkqkn6h2sia8dcsi72wbcfl-vm-test-run-unnamed
This can be also done with one command:
result=$(readlink -f ./result) rm ./result && nix-store --delete $result
Tests that need multiple virtual machines#
Tests can involve multiple virtual machines.
This example uses the use-case of a REST interface to a PostgreSQL database. The following example Nix expression is adapted from How to use NixOS for lightweight integration tests.
This tutorial follows PostgREST tutorial, a generic RESTful API for PostgreSQL.
If you skim over the official tutorial, you’ll notice there’s quite a bit of setup in order to test if all the steps work.
The setup includes:
A virtual machine named
server
running PostgreSQL and PostgREST.A virtual machine named
client
running HTTP client queries usingcurl
.A
testScript
orchestrating testing logic betweenclient
andserver
.
The complete postgrest.nix
file looks like the following:
let
# Pin Nixpkgs, as some packages are broken in the 22.11 release
nixpkgs = fetchTarball "https://github.com/NixOS/nixpkgs/archive/0f8f64b54ed07966b83db2f20c888d5e035012ef.tar.gz";
pkgs = import nixpkgs { config = {}; overlays = []; };
# Single source of truth for all tutorial constants
database = "postgres";
schema = "api";
table = "todos";
username = "authenticator";
password = "mysecretpassword";
webRole = "web_anon";
postgrestPort = 3000;
# NixOS module shared between server and client
sharedModule = {
# Since it's common for CI not to have $DISPLAY available, explicitly disable graphics support
virtualisation.graphics = false;
};
in
pkgs.nixosTest {
# NixOS tests are run inside a virtual machine, and here you specify its system type
system = "x86_64-linux";
name = "postgres-test";
nodes = {
server = { config, pkgs, ... }: {
imports = [ sharedModule ];
networking.firewall.allowedTCPPorts = [ postgrestPort ];
services.postgresql = {
enable = true;
initialScript = pkgs.writeText "initialScript.sql" ''
create schema ${schema};
create table ${schema}.${table} (
id serial primary key,
done boolean not null default false,
task text not null,
due timestamptz
);
insert into ${schema}.${table} (task) values ('finish tutorial 0'), ('pat self on back');
create role ${webRole} nologin;
grant usage on schema ${schema} to ${webRole};
grant select on ${schema}.${table} to ${webRole};
create role ${username} inherit login password '${password}';
grant ${webRole} to ${username};
'';
};
users = {
mutableUsers = false;
users = {
# For ease of debugging the VM as the `root` user
root.password = "";
# Create a system user that matches the database user so that you
# can use peer authentication. The tutorial defines a password,
# but it's not necessary.
"${username}".isSystemUser = true;
};
};
systemd.services.postgrest = {
wantedBy = [ "multi-user.target" ];
after = [ "postgresql.service" ];
script =
let
configuration = pkgs.writeText "tutorial.conf" ''
db-uri = "postgres://${username}:${password}@localhost:${toString config.services.postgresql.port}/${database}"
db-schema = "${schema}"
db-anon-role = "${username}"
'';
in "${pkgs.haskellPackages.postgrest}/bin/postgrest ${configuration}";
serviceConfig.User = username;
};
};
client = {
imports = [ sharedModule ];
};
};
# Disable linting for simpler debugging of the testScript
skipLint = true;
testScript = ''
import json
import sys
start_all()
server.wait_for_open_port(${toString postgrestPort})
expected = [
{"id": 1, "done": False, "task": "finish tutorial 0", "due": None},
{"id": 2, "done": False, "task": "pat self on back", "due": None},
]
actual = json.loads(
client.succeed(
"${pkgs.curl}/bin/curl http://server:${toString postgrestPort}/${table}"
)
)
assert expected == actual, "table query returns expected content"
'';
}
Unlike the previous example, the virtual machines need an expressive name to distinguish them.
For this example we choose client
and server
.
Set up all machines and run the test script:
nix-build postgrest.nix
...
test script finished in 10.96s
cleaning up
killing client (pid 10)
killing server (pid 22)
(0.00 seconds)
/nix/store/bx7z3imvxxpwkkza10vb23czhw7873w2-vm-test-run-unnamed
## Additional information regarding NixOS tests:
- Running integration tests on CI requires hardware acceleration, which many CIs do not support.
To run integration tests on [GitHub Actions](<github-actions>) see [how to disable hardware acceleration](https://github.com/cachix/install-nix-action#how-do-i-run-nixos-tests).
- NixOS comes with a large set of tests that serve also as educational examples.
A good inspiration is [Matrix bridging with an IRC](https://github.com/NixOS/nixpkgs/blob/master/nixos/tests/matrix/appservice-irc.nix).
<!-- TODO: move examples from https://nixos.wiki/wiki/NixOS_Testing_library to the NixOS manual and troubleshooting tips to nix.dev -->