Run a reconstruction with MPI

The MPI programs spread one dataset's reconstruction over several processes, which can be on different nodes. This page shows how to build and run them. The command-line reference lists their options.

Build the MPI programs

Install an MPI implementation and Boost.MPI (for example libboost-mpi-dev and libopenmpi-dev on Ubuntu). MPISupport is on by default, so a plain configure builds the MPI programs for every backend you enable, as long as CMake finds MPI:

cmake -B build -DCUDA_COMPUTE=ON
cmake --build build -j$(nproc)
cmake --install build --prefix $HOME/forge

They install next to the other programs, in bin/<backend>/.

Choose a program

Program	Use it for	How it splits the work
forgeSenseMPI	forgeSense reconstructions with many images	Images (slices, repetitions, averages, echoes, phases) are dealt round-robin to the ranks. Each rank reconstructs and writes its own.
forgePcSenseMPI	One large forgePcSense image at a time	All ranks work on each image together: the operator's shot-and-coil pairs are dealt round-robin. Rank 0 writes the output.
forgePcSenseMPI_TS	forgePcSenseTimeSeg reconstructions with many images	Images are dealt round-robin, as in forgeSenseMPI

Run it

mpirun -np 4 forgeSenseMPI --no-tui \
    -i data.h5 -o recon/ -F NUFFT -I minmax -t 6

• forgeSenseMPI needs -I and -t; the single-process forgeSense does not.
• Pass --no-tui so the ranks don't each try to start forgeview.
• Only rank 0 logs to the terminal. Add --mpi-log-all to see every rank.

Use several GPUs

In CUDA builds, --device auto (the default) gives rank r GPU r mod (number of GPUs), so starting as many ranks per node as there are GPUs puts one rank on each:

mpirun -np 4 forgeSenseMPI --no-tui -i data.h5 -o recon/ -F NUFFT -I minmax -t 6

To put every rank on one GPU, pass its index, for example --device 1. CPU and Metal builds ignore --device.