Create an MPI-IO File

In this section we are going to create an MPI file. It is going to be an HDF5 file too, so it won't be empty, even though we are not going to write anything on it explicitly yet. The file is going to be opened by all processes in the MPI_COMM_WORLD communicator.

Here is the program taken from the NCSA HDF5 Tutorial, Creating/Accessing a File with PHDF5

/*  
 *  This example creates an HDF5 file.
 */
 
#include "hdf5.h"

#define H5FILE_NAME     "SDS_row.h5"

int
main (int argc, char **argv)
{
    /*
     * HDF5 APIs definitions
     */ 	
    hid_t       file_id;         /* file and dataset identifiers */
    hid_t	plist_id;        /* property list identifier( access template) */
    herr_t	status;

    /*
     * MPI variables
     */
    int mpi_size, mpi_rank;
    MPI_Comm comm  = MPI_COMM_WORLD;
    MPI_Info info  = MPI_INFO_NULL;

    /*
     * Initialize MPI
     */
    MPI_Init(&argc, &argv);
    MPI_Comm_size(comm, &mpi_size);
    MPI_Comm_rank(comm, &mpi_rank);  
 
    /* 
     * Set up file access property list with parallel I/O access
     */
     plist_id = H5Pcreate(H5P_FILE_ACCESS);
     H5Pset_fapl_mpio(plist_id, comm, info);

    /*
     * Create a new file collectively.
     */
    file_id = H5Fcreate(H5FILE_NAME, H5F_ACC_TRUNC, H5P_DEFAULT, plist_id);

    /*
     * Close property list.
     */
    H5Pclose(plist_id);

    /*
     * Close the file.
     */
    H5Fclose(file_id);
 
    MPI_Finalize();

    return 0;
}

Observe that the program does not include <mpi.h> explicitly. This is because it is included by <hdf5.h> through <H5public.h>. We compile this program with h5cc, which invokes mpicc internally:

gustav@bh1 $ h5cc -o MPI_file_create MPI_file_create.c
gustav@bh1 $ mv MPI_file_create ~/bin/MPI_file_create

After compilation and linking I have moved the binary to my ~/bin so that it will be available on other nodes when I run it under mpiexec. Then I start the MPD machine to run the program:

gustav@bh1 $ mpdboot
gustav@bh1 $ mpdtrace | wc
     26      26     129
gustav@bh1 $

and go to GPFS to run it:

gustav@bh1 $ cd /N/gpfs/gustav
gustav@bh1 $ mpiexec -n 8 MPI_file_create
gustav@bh1 $ ls
SDS_row.h5  t_mpio_1wMr  test  tests
gustav@bh1 $

There are various other files and directories in this area, but SDS_row.h5 has been created and it is a genuine HDF5 file as you can see if you run h5dump on it:

gustav@bh1 $ h5dump SDS_row.h5
HDF5 "SDS_row.h5" {
GROUP "/" {
}
}
gustav@bh1 $

Now let us have a look at the program itself. This is a typical simple MPI program, which begins with the usual:

    MPI_Init(&argc, &argv);
    MPI_Comm_size(comm, &mpi_size);
    MPI_Comm_rank(comm, &mpi_rank);

But right after this we invoke HDF5 functions. First we create a default file access list:

     plist_id = H5Pcreate(H5P_FILE_ACCESS);

and then we invoke function  H5Pset_fapl_mpio:

    H5Pset_fapl_mpio(plist_id, comm, info);

which is going to associate the file with the MPI_COMM_WORLD communicator and with the MPI_INFO_NULL info object, because we have declared that:

    MPI_Comm comm  = MPI_COMM_WORLD;
    MPI_Info info  = MPI_INFO_NULL;

Now we create the file itself:

    file_id = H5Fcreate(H5FILE_NAME, H5F_ACC_TRUNC, H5P_DEFAULT, plist_id);

and this is all we are going to do in this program. Having created the file, we close the property list, then close the file itself and finally we close MPI with the call to MPI_Finalize - remember that this is an MPI program after all:

    H5Pclose(plist_id);
    H5Fclose(file_id);
    MPI_Finalize();
    return 0;
}