Guix-HPC Activity Report, 2023

We have some exciting news to share: AMD has just contributed 100+ Guix packages adding several versions of the whole HIP and ROCm stack! ROCm is AMD’s Radeon Open Compute Platform, a set of low-level support tools for general-purpose computing on graphics processing units (GPGPUs), and HIP is the Heterogeneous Interface for Portability, a language one can use to write code (computational kernels) targeting GPUs or CPUs. The whole stack is free and “open source” software—a breath of fresh air!—and is seeing increasing adoption in HPC. And, it can now be deployed with Guix!

Continuous integration and continuous delivery for HPC

Ludovic Courtès — March 6, 2023

Sysadmin

Will those binaries actually work? This is a central question for HPC practitioners and one that’s sometimes hard to answer: increasingly complex software stacks being deployed, and often on a variety of clusters. Will that program pick the right libraries? Will it perform well? With each cluster having its own hardware characteristics, portability is often considered unachievable. As a result, HPC practitioners rarely take advantage of continuous integration and continuous delivery (CI/CD): building software locally on the cluster is common, and software validation is often a costly manual process that has to be repeated on each cluster.

Guix-HPC Activity Report, 2022

Céline Acary-Robert, Ludovic Courtès, Yann Dupont, Marek Felšöci, Konrad Hinsen, Ontje Lünsdorf, Pjotr Prins, Philippe Swartvagher, Simon Tournier, Ricardo Wurmus — February 10, 2023

This document is also available as PDF (printable booklet).

Back to the future: modules for Guix packages

Ludovic Courtès — May 6, 2022

Sysadmin

Modules

Releases

Some things in our software world are timeless. The venerable Environment Modules are one of these. If you’ve ever used a high-performance cluster in the last three decades, chances are you’re already familiar with it. Modules is about managing software environments, just like Guix is—or, perhaps more accurately, guix shell.

Guix-HPC Activity Report, 2021

Pierre-Antoine Bouttier, Ludovic Courtès, Yann Dupont, Marek Felšöci, Felix Gruber, Konrad Hinsen, Arun Isaac, Pjotr Prins, Philippe Swartvagher, Simon Tournier, Ricardo Wurmus — February 3, 2022

This document is also available as PDF (printable booklet).

Tuning packages for a CPU micro-architecture

Ludovic Courtès — January 6, 2022

Performance

Optimization

It should come as no surprise that the execution speed of programs is a primary concern in high-performance computing (HPC). Many HPC practitioners would tell you that, among their top concerns, is the performance of high-speed networks used by the Message Passing Interface (MPI) and use of the latest vectorization extensions of modern CPUs.

Guix-HPC Activity Report, 2020

Lars-Dominik Braun, Ludovic Courtès, Pjotr Prins, Simon Tournier, Ricardo Wurmus — February 9, 2021

This document is also available as PDF (printable booklet).

Faster relocatable packs with Fakechroot

Ludovic Courtès — May 18, 2020

Containers

The guix pack command creates “application bundles” that can be used to deploy software on machines that do not run Guix (yet!), such as HPC clusters. Since its inception in 2017, it has seen a number of improvements, such as the ability to create Docker and Singularity container images. Some clusters lack these tools, though, and the addition of relocatable packs was a way to address that. This post looks at a new execution engine for relocatable packs that has just landed with the goal of improving performance.

Guix-HPC Activity Report, 2019

Ludovic Courtès, Paul Garlick, Konrad Hinsen, Pjotr Prins, Ricardo Wurmus — February 17, 2020

This document is also available as PDF (printable booklet).

Optimized and portable Open MPI packaging

Ludovic Courtès — December 19, 2019

Performance

Optimization

High-performance networks have constantly been evolving, in sometimes hard-to-decipher ways. Once upon a time, hardware vendors would pre-install an MPI implementation (often an in-house fork of one of the free MPI implementations) specially tailored for their hardware. Fortunately, this time appears to be gone. Despite that, there is still widespread belief that MPI cannot be packaged in a way that achieves best performance on a variety of contemporary high-speed networking hardware.

Guix-HPC Activity Report, 2018

Eric Bavier, Ludovic Courtès, Paul Garlick, Pjotr Prins, Ricardo Wurmus — February 12, 2019

This document is also available as PDF (printable booklet).

PiGx paper awarded at the International Conference on Genomics (ICG-13)

Ricardo Wurmus — January 11, 2019

December 2018 the Akalin lab at the Berlin Institute of Medical Systems Biology (BIMSB) published a paper about a collection of reproducible genomics pipelines called PiGx that are made available through GNU Guix. The article was awarded third place in the GigaScience ICG-13 Prize. Representing the authors, Ricardo Wurmus was invited to present the work on PiGx and Guix in Shenzhen, China at ICG-13.

Ricardo urged the audience of wet lab scientists and bioinformaticians to apply the same rigorous standards of experimental design to experiments involving software: all variables need to be captured and constrained. To demonstrate that this does not need to be complicated, Ricardo reported the experiences of the Akalin lab in building a collection of reproducibly built automated genomics workflows using GNU Guix.

Due to technical difficulties the recording of the talk was lost, so Ricardo re-recorded the talk a few weeks later.

Paper on reproducible bioinformatics pipelines with Guix

Ricardo Wurmus — May 9, 2018

Reproducibility

Bioinformatics

I’m happy to announce that the bioinformatics group at the Max Delbrück Center that I’m working with has released a preprint of a paper on reproducibility with the title Reproducible genomics analysis pipelines with GNU Guix.

Guix-HPC debut!

Ludovic Courtès, Roel Janssen, Pjotr Prins, Ricardo Wurmus — September 5, 2017

Reproducibility