Integration testing using virtual machines (VMs)
Contents
Integration testing using virtual machines (VMs)¶
One of the most powerful features in the Nix ecosystem is the ability to provide a set of declarative NixOS configurations and use a simple Python interface to interact with them using QEMU as the backend.
Those tests are widely used to ensure that NixOS works as intended, so in general they are called NixOS tests. They can be written and launched outside of NixOS, on any Linux machine (with MacOS support coming soon).
Integration tests are reproducible due to the design properties of Nix, making them a valuable part of a Continuous Integration (CI) pipeline.
Testing a typical web application backed by PostgreSQL¶
This tutorial follows PostgREST tutorial, a generic RESTful API for PostgreSQL.
If you skim over the official tutorial, you’ll notice there’s quite a bit of setup in order to test if all the steps work.
We are going to set up:
A VM named
server
running postgreSQL and postgREST.A VM named
client
running HTTP client queries usingcurl
.A
testScript
orchestrating testing logic betweenclient
andserver
.
The following example Nix expression is adapted from How to use NixOS for lightweight integration tests.
Writing the test¶
Create postgrest.nix
:
1let
2 # Pin nixpkgs, see pinning tutorial for more details
3 nixpkgs = fetchTarball "https://github.com/NixOS/nixpkgs/archive/0f8f64b54ed07966b83db2f20c888d5e035012ef.tar.gz";
4 pkgs = import nixpkgs {};
5
6 # Single source of truth for all tutorial constants
7 database = "postgres";
8 schema = "api";
9 table = "todos";
10 username = "authenticator";
11 password = "mysecretpassword";
12 webRole = "web_anon";
13 postgrestPort = 3000;
14
15 # NixOS module shared between server and client
16 sharedModule = {
17 # Since it's common for CI not to have $DISPLAY available, we have to explicitly tell the tests "please don't expect any screen available"
18 virtualisation.graphics = false;
19 };
20
21in pkgs.nixosTest ({
22 # NixOS tests are run inside a virtual machine, and here we specify system of the machine.
23 system = "x86_64-linux";
24
25 nodes = {
26 server = { config, pkgs, ... }: {
27 imports = [ sharedModule ];
28
29 networking.firewall.allowedTCPPorts = [ postgrestPort ];
30
31 services.postgresql = {
32 enable = true;
33
34 initialScript = pkgs.writeText "initialScript.sql" ''
35 create schema ${schema};
36
37 create table ${schema}.${table} (
38 id serial primary key,
39 done boolean not null default false,
40 task text not null,
41 due timestamptz
42 );
43
44 insert into ${schema}.${table} (task) values ('finish tutorial 0'), ('pat self on back');
45
46 create role ${webRole} nologin;
47 grant usage on schema ${schema} to ${webRole};
48 grant select on ${schema}.${table} to ${webRole};
49
50 create role ${username} inherit login password '${password}';
51 grant ${webRole} to ${username};
52 '';
53 };
54
55 users = {
56 mutableUsers = false;
57 users = {
58 # For ease of debugging the VM as the `root` user
59 root.password = "";
60
61 # Create a system user that matches the database user so that we
62 # can use peer authentication. The tutorial defines a password,
63 # but it's not necessary.
64 "${username}".isSystemUser = true;
65 };
66 };
67
68 systemd.services.postgrest = {
69 wantedBy = [ "multi-user.target" ];
70 after = [ "postgresql.service" ];
71 script =
72 let
73 configuration = pkgs.writeText "tutorial.conf" ''
74 db-uri = "postgres://${username}:${password}@localhost:${toString config.services.postgresql.port}/${database}"
75 db-schema = "${schema}"
76 db-anon-role = "${username}"
77 '';
78 in "${pkgs.haskellPackages.postgrest}/bin/postgrest ${configuration}";
79 serviceConfig.User = username;
80 };
81 };
82
83 client = {
84 imports = [ sharedModule ];
85 };
86 };
87
88 # Disable linting for simpler debugging of the testScript
89 skipLint = true;
90
91 testScript = ''
92 import json
93 import sys
94
95 start_all()
96
97 server.wait_for_open_port(${toString postgrestPort})
98
99 expected = [
100 {"id": 1, "done": False, "task": "finish tutorial 0", "due": None},
101 {"id": 2, "done": False, "task": "pat self on back", "due": None},
102 ]
103
104 actual = json.loads(
105 client.succeed(
106 "${pkgs.curl}/bin/curl http://server:${toString postgrestPort}/${table}"
107 )
108 )
109
110 assert expected == actual, "table query returns expected content"
111 '';
112})
A few notes:
Between the machines defined inside the
nodes
attribute, hostnames are resolved based on their attribute names. In this case we haveclient
andserver
.The testing framework exposes a wide set of operations used inside the
testScript
. A full set of testing operations is part of VM testing operations API Reference.
Running tests¶
To set up all machines and execute the test script:
$ nix-build postgrest.nix
You’ll notice an error message if something goes wrong.
In case the tests succeed, you should see at the end:
...
test script finished in 10.96s
cleaning up
killing client (pid 10)
killing server (pid 22)
(0.00 seconds)
/nix/store/bx7z3imvxxpwkkza10vb23czhw7873w2-vm-test-run-unnamed
Developing and debugging tests¶
When developing tests or when something breaks, it’s useful to interactively fiddle with the script or access a terminal for a machine.
To interactively start a Python session with a testing framework:
$ $(nix-build -A driverInteractive postgrest.nix)/bin/nixos-test-driver
...
starting VDE switch for network 1
>>>
You can run any of the testing operations.
The testScript
attribute from our postgrest.nix
definition can be executed with test_script()
function.
To start all machines and enter a telnet terminal to a specific machine:
>>> start_all()
...
>>> server.shell_interact()
server: Terminal is ready (there is no prompt):
uname -a
Linux server 5.10.37 #1-NixOS SMP Fri May 14 07:50:46 UTC 2021 x86_64 GNU/Linux
Next steps¶
Running integration tests on CI requires hardware acceleration, which many CIs do not support. To run integration tests on GitHub Actions see how to disable hardware acceleration.
NixOS comes with a large set of tests that serve also as educational examples. A good inspiration is Matrix bridging with an IRC.