Reproducibility vs. root privileges
Guix is a good fit for multi-user environments such as clusters:
allows non-root users to install packages at will without interfering with each other.
However, a common complaint is that installing Guix requires administrator
privileges. More precisely,
guix-daemon, the system-wide daemon that
spawns package builds and downloads on behalf of
must be running as
This is not much of a problem on one's laptop but it surely makes it
harder to adopt Guix on an HPC cluster.
So why does Guix have this requirement when other tools don’t? In this article we look at the various options available today to achieve build isolation—a prerequisite for reproducible builds—on GNU/Linux, and how HPC software deployment tools address the problem.
Creating isolated environments
GNU Guix prides itself on being able to create isolated build environments, which in turn helps make sure that package builds are reproducible (the same inputs yield the same output) regardless of what software is installed on the machine or what machine performs the build. It does that in the traditional Unix way, which we'll describe below.
Unix-like operating systems have traditionally provided a couple of
tools to isolate processes:
allows a process to “see” a different root file system, and
allows a process to run on behalf of a different user.
guix-daemon uses these two mechanisms when it builds something: it
chroots into an environment where only the declared dependencies of a
build process are accessible, and it
setuids to a specific “build
user” that does not run any process other than this one build process.
For this reason, the manual instructs
to create a pool of build users upfront.
These two mechanisms are all it takes to achieve process isolation. Of course processes still run under the same operating system kernel as before, unlike what a virtual machine would provide, but the rest is unshared. Today this is the only portable way to achieve process isolation on a POSIX system.
guix-daemon runs processes in
separate PID, networking, and mount “namespaces”.
a feature of the kernel Linux
to improve process isolation. For example, a process running under a
separate PID namespace has a different view of the existing set of
process IDs; it cannot reference a process running in a separate
The problem of
setuid, and namespaces is that they are
available only to
root. A few years ago, the kernel Linux gained
which hold the promise of providing unprivileged users with a way to
isolate processes. Unfortunately, user namespaces are still disabled by
for fear of security issues—and
I should say rightfully so if we look, for example,
this May 2017
PF_PACKET vulnerability in the kernel,
which was exploitable by unprivileged users in a user namespace.
Hopefully this will be fixed in the not-too-distant future, but for the time being, this is not a feature we can expect to find on HPC clusters, on which we want to install Guix.
When everything else fails
When none of the above is available, the remaining option to achieve
process isolation is PRoot. PRoot
is a program that runs your application and uses the
system call to intercept all its system calls and, if permitted,
“translates” them into an equivalent system call in the “host”
For example, PRoot can do file system virtualization akin to
and bind mounts.
To do that, it needs to intercept
open calls, and translate file names
in the isolated environments to file names outside the environment—or
raise an error when trying to access files that are not mapped into the
The downside of this is performance: intercepting and translating system calls is costly. On the other hand, a mostly-computational application such as a long-running numerical simulation will be largely unaffected by this overhead.
In a future post we will see how to take advantage of PRoot in
What do others do?
In the context of HPC software deployment, people have been looking at ways to achieve reproducibility, and also to avoid requiring root privileges. As we’ve seen above, it’s usually a tradeoff that must be made.
EasyBuild and Spack
EasyBuild and Spack, two package managers designed for HPC clusters, have the advantage of not requiring root privileges at all. Thus, provided Python is installed on the cluster you want to use, you can readily install them and use them to build the packages they provide.
This advantage comes at the cost of reproducibility. Build processes are not isolated from the rest of the system, so they can pick and choose software from the host distribution. The distributions of Spack and EasyBuild are actually not self-contained: they assume that some specific packages are available on the host system, such as a C compiler or the GNU Binutils.
This leads to very concrete reproducibility issues, where things might build on one machine and fail to build on another, simply because the core software packages differ.
Singularity is a tool to build and create Docker-style application bundles (sometimes confusingly referred to as “containers”). To run those application bundles, it needs at the very least file system virtualization—it needs to “map” file names within the image at the same place in the execution environment.
Singularity’s web site explains that no root-owned daemon processes are required on the HPC cluster where it is used. However, it also notes that it needs either a setuid-root helper program to create isolated environments on behalf of users, or support for user namespaces. In practice, that means that only cluster admins can install it today.
Shifter relies on the Docker daemon to execute application bundles, which needs to be installed as root.
runc prides itself on having “the ability to run containers without root privileges”, which they call rootless containers. There is no magic here: its implementation simply requires support for user namespaces.
Guix requires a root-owned daemon to perform isolated builds, which are the foundation for reproducible software environments. This makes it less readily available to HPC cluster users: you have to convince your system administrators to install it before you can happily use it to manage your software (here’s a trick: tell them that’ll give you more flexibility and also relieve them from the tedious manual management of environment modules :-)).
However, the kernel Linux does not yet provide a mechanism for
non-root users to build isolated environments. In the future, when user
namespaces are widely available, the problem will be solved. But for
now, if you value reproducibility, let’s talk to cluster sysadmins and
invite them to install
But wait, we also have a solution for you in the meantime. More on that in a future post. :-)