6.8 FFTW MPI Wisdom

FFTW’s “wisdom” facility (see Words of Wisdom-Saving Plans) can be used to save MPI plans as well as to save uniprocessor plans. However, for MPI there are several unavoidable complications.

First, the MPI standard does not guarantee that every process can perform file I/O (at least, not using C stdio routines)—in general, we may only assume that process 0 is capable of I/O.⁷ So, if we want to export the wisdom from a single process to a file, we must first export the wisdom to a string, then send it to process 0, then write it to a file.

Second, in principle we may want to have separate wisdom for every process, since in general the processes may run on different hardware even for a single MPI program. However, in practice FFTW’s MPI code is designed for the case of homogeneous hardware (see Load balancing), and in this case it is convenient to use the same wisdom for every process. Thus, we need a mechanism to synchronize the wisdom.

To address both of these problems, FFTW provides the following two functions:

void fftw_mpi_broadcast_wisdom(MPI_Comm comm);
void fftw_mpi_gather_wisdom(MPI_Comm comm);

Given a communicator comm, fftw_mpi_broadcast_wisdom will broadcast the wisdom from process 0 to all other processes. Conversely, fftw_mpi_gather_wisdom will collect wisdom from all processes onto process 0. (If the plans created for the same problem by different processes are not the same, fftw_mpi_gather_wisdom will arbitrarily choose one of the plans.) Both of these functions may result in suboptimal plans for different processes if the processes are running on non-identical hardware. Both of these functions are collective calls, which means that they must be executed by all processes in the communicator.

So, for example, a typical code snippet to import wisdom from a file and use it on all processes would be:

{
    int rank;

    fftw_mpi_init();
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);
    if (rank == 0) fftw_import_wisdom_from_filename("mywisdom");
    fftw_mpi_broadcast_wisdom(MPI_COMM_WORLD);
}

(Note that we must call fftw_mpi_init before importing any wisdom that might contain MPI plans.) Similarly, a typical code snippet to export wisdom from all processes to a file is:

{
    int rank;

    fftw_mpi_gather_wisdom(MPI_COMM_WORLD);
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);
    if (rank == 0) fftw_export_wisdom_to_filename("mywisdom");
}