MPI software stacks are complex beasts that play a key role in application performance. Guix packages Open MPI for a wide range high-speed interconnects—InfiniBand, Omni-Path, Slingshot, ROCE, you name it—with the goal of achieving peak performance for each one of them.

The key challenge from a packaging perspective is to ensure no performance regression occur for any of these network types when Open MPI or its dependencies change. Administrators of clusters where Guix is the main deployment tool typically have checks in place for this. But what if a regression does occur?

This blog post is a report from the trenches, showing how a performance regression of the MPI stack on Slingshot interconnects was identified. The debugging story is in itself a showcase for the control and transparency that Guix affords.

The plot

Slingshot support is relatively recent in Open MPI as packaged by Guix and, until now, concerned a relatively small part of Guix usage in HPC. No automated performance regression testing was in place for this interconnect, primarily due to difficulties accessing hardware that would allow us to do that. And so, the inevitable happened: a performance regression struck.

The symptom is simple: running an MPI bandwidth/latency benchmark would show poor performance (here we force the use of version 5 of Open MPI with --with-input):

$ guix shell intel-mpi-benchmarks --with-input=openmpi=openmpi@5 openmpi@5 -- \
    mpirun IMB-MPI1 PingPong
…
# PingPong

#---------------------------------------------------
# Benchmarking PingPong 
# #processes = 2 
#---------------------------------------------------
       #bytes #repetitions      t[usec]   Mbytes/sec
            0         1000        37.94         0.00
            1         1000        38.30         0.03
            2         1000        38.41         0.05
…
      1048576           40       575.85      1820.92
      2097152           20       983.58      2132.15
      4194304           10      1817.18      2308.15

Uh-oh—peak bandwidth is 2.3 GB/s instead the expected 25 GB/s or so.

The reason this happens is because libcxi (the low-level Slingshot support library) errors out, and then libfabric (which sits right above libcxi) propagates that error to Open MPI, which in turn falls back to plain TCP/IP over Ethernet.

We can get debugging info from Open MPI, libfabric, and libcxi by passing the right flags and environment variables:

$ FI_LOG_LEVEL=9 guix shell intel-mpi-benchmarks --with-input=openmpi=openmpi@5 openmpi@5 -- mpirun --mca btl_ofi_verbose 3 --mca pml_ucx_verbose 3  --mca btl ofi --mca mtl ofi --mca btl_base_verbose 100 IMB-MPI1 PingPong
  [c1510:1386001] mca: base: components_register: registering framework btl components
  [c1510:1386001] mca: base: components_register: found loaded component ofi
  [c1510:1386001] mca: base: components_register: component ofi register function successful
  …
  [c1510:1386001] mtl_ofi_component.c:343: mtl:ofi:provider: cxi
  [c1510:1386001] mtl_ofi_component.c:368: mtl:ofi:provider:domain: cxi0
  libfabric:1386001:1781018439::cxi:domain:cxip_domain_enable():457<warn> c1510: Security Issue: Using unrestricted service ID 1 for cxi0. Please provide a service ID via auth_key fields.
  libfabric:1386001:1781018439::cxi:domain:cxip_cp_get():109<warn> c1510: Failed to allocate CP, ret: -22 VNI: 1 TC: BEST_EFFORT TYPE: DEFAULT
  libfabric:1386001:1781018439::cxi:domain:cxip_cp_get():112<warn> c1510: Remapping original TC from BEST_EFFORT to BEST_EFFORT
  libfabric:1386001:1781018439::cxi:domain:cxip_cp_get():137<warn> c1510: Failed to allocate CP, ret: -22 VNI: 1 TC: BEST_EFFORT TYPE: DEFAULT
  libfabric:1386001:1781018439::cxi:domain:cxip_cmdq_alloc():289<warn> c1510: Failed to allocate CP: -22
  libfabric:1386001:1781018439::cxi:ep_ctrl:cxip_ep_cmdq():95<warn> c1510: Unable to allocate CMDQ, ret: -22
  libfabric:1386001:1781018439::cxi:ep_ctrl:cxip_ep_ctrl_init():567<warn> c1510: Failed to allocate control TXQ, ret: -28
  libfabric:1386001:1781018439::cxi:ep_ctrl:cxip_ep_enable():691<warn> c1510: cxip_ep_ctrl_init returned: -262
  [c1510:1386001] mtl_ofi_component.c:1093:fi_enable failed: Invalid resource domain
  [c1510:1386001] select: initializing btl component ofi
  [c1510:1386001] btl_ofi_component.c:318: btl:ofi:provider_include = "(null)"
  [c1510:1386001] btl_ofi_component.c:320: btl:ofi:provider_exclude = "shm,sockets,tcp,udp,rstream,usnic,net"
  libfabric:1386001:1781018439::core:core:fi_getinfo_():1391<warn> fi_getinfo: provider efa output empty list

The main clue here are the “Failed to allocate CP” errors that originate in the Slingshot “provider” of libfabric (CP stands for communication profile in libcxi).

Bisecting the distro

Where do we go from here? Short of being a libcxi expert, a good way forward is to find out what change introduced that regression. There are two good news here: first the MPI stack provided by Guix is entirely self-contained—so software on the machine that is not provided by Guix is unlikely to be the culprit—, and second we knew of a known-good state.

More precisely, we knew the exact commit of Guix that provided a “good” MPI stack for Slingshot; and we knew of a commit that provided a “bad” MPI stack. It is worth stressing this: the Guix commit shown by guix describe is all it takes to identify the entire MPI stack—package versions, configure flags, compilers, etc. From a Guix commit, we can redeploy that stack any time with guix time-machine.

Naturally, we bisected Guix itself in search of the offending commit. At each step, we allocate two nodes and run:

guix time-machine -q --commit=THE-COMMIT -- \
  shell intel-mpi-benchmarks --with-input=openmpi=openmpi@5 openmpi@5 -- \
  mpirun IMB-MPI1 PingPong

In a matter of minutes, we’d be able to categorize the commit being tested as “good” or “bad”. (Automating categorization by parsing the output of the benchmark would have been nice but in this case the cost/benefit ratio did not look favorable.)

The offending commit turned out to be an innocent-looking libcxi upgrade.

Finer-grain bisection

Why stop at the distro level when we could pinpoint the libcxi commit that introduces the regression? While time-machine allows us to test specific changes in the distro, the --with-commit option allows us to rebuild an entire stack with a specific commit of one of its packages—keeping the rest of the stack unchanged.

Using this feature we were able to continue our bug hunt, this time bisecting at the level of individual libcxi commits, starting with the known-good commit (corresponding to libcxi version 13.0.0) up to the first known-bad commit (libcxi version 13.1.0):

guix shell intel-mpi-benchmarks --with-input=openmpi=openmpi@5 openmpi@5 \
  --with-commit=libcxi=THE-COMMIT -- \
  mpirun IMB-MPI1 PingPong

This bisection is a bit more expensive than the previous one because at each step, most of the MPI stack needs to be rebuilt (specifically libcxi and its dependents: libfabric and Open MPI). Still, after a few hours, the offending commit was identified, which allowed us to provide a detailed bug report for the libcxi developers (the bug is still opened as of this writing).

Conclusion

The reason I write about this anecdote is because I think it demonstrates a kind of control over the software stack that is unthinkable without Guix or its sibling Nix. With guix time-machine we were able to bisect the entire distro until we could find the commit that introduced the regression; using --with-commit we continued that investigation at the level of a single package.

There is currently no guix bisect command, but it sure sounds like it would be a useful one to have, as this story demonstrates.

As for tracking MPI stack performance, we are looking forward to setting up a publicly accessible dashboard tracking Open MPI performance on supported interconnects using the Intel or OSU micro-benchmarks. Stay tuned!