The Discussion

The beginning of the program is the same as in mkrandfiles until we get to the place where processes have to allocate space for data. This is done the same way we did it in mkrandpfile, i.e., we allocate enough space to generate all the data and then to write it in one shove onto the file:

    /* Allocate space needed to generate the integers */

    number_of_integers = number_of_blocks * BLOCK_SIZE;
    number_of_bytes = sizeof(int) * number_of_integers;
    total_number_of_integers = (long long) number_of_integers
      * (long long) pool_size;
    total_number_of_bytes = (long long) number_of_bytes
      * (long long) pool_size;
    junk = (int*) malloc(number_of_bytes);

Now every process generates its own file name:

    sprintf(file_name, "%s.%d", basename, my_rank);

#ifdef DEBUG
    printf("%3d: opening file %s\n", my_rank, file_name);
#endif

and opens the file:

    my_file_open_error =
      MPI_File_open(MPI_COMM_SELF, file_name, 
                    MPI_MODE_CREATE | MPI_MODE_WRONLY, MPI_INFO_NULL, &fh);

Observe the use of the predefined  MPI_COMM_SELF communicator. This is a communicator that contains one process only, namely the process that uses it. Consequently the file in question will not be shared with other processes. It belongs to this process in entirety.

Now we check for possible errors on open:

    if (my_file_open_error != MPI_SUCCESS) {

      MPI_Error_class(my_file_open_error, &error_class);
      MPI_Error_string(error_class, error_string, &length_of_error_string);
      printf("%3d: %s\n", my_rank, error_string);

      MPI_Error_string(my_file_open_error, error_string, 
                       &length_of_error_string);
      printf("%3d: %s\n", my_rank, error_string);

      my_file_open_error = TRUE;

    }

We don't call MPI_Abort. Instead we set my_file_open_error to TRUE. If MPI_SUCCESS was zero, then we could use the value of my_file_open_error simply as returned by MPI_File_open, but MPI_SUCCESS is opaque and we must not make such assumptions.

Now all processes check if any of them experienced problems opening the file:

    MPI_Allreduce (&my_file_open_error, &file_open_error, 1, MPI_INT, 
                   MPI_LOR, MPI_COMM_WORLD);

#ifdef DEBUG
    if (i_am_the_master)
      if (file_open_error)
        fprintf(stderr, "problem opening output files\n");
#endif

The remaining part of the program is one large if statement followed by cleanup that depends on what happened before:

    if (! file_open_error) {
       blah... blah... blah...
    }

    if (!my_file_open_error) MPI_File_close(&fh);

    if (write_error || file_open_error) {
      if (! my_file_open_error) MPI_File_delete(file_name, MPI_INFO_NULL);
    }
    else {
      MPI_Reduce(&io_time, &longest_io_time, 1, MPI_DOUBLE, MPI_MAX,
                 MASTER_RANK, MPI_COMM_WORLD);
      if (i_am_the_master) {
        printf("longest_io_time       = %f seconds\n", longest_io_time);
        printf("total_number_of_bytes = %lld\n", total_number_of_bytes);
        printf("transfer rate         = %f MB/s\n", 
               total_number_of_bytes / longest_io_time / MBYTE);
      }
    }

Here you see that in case of a problem all successfully opened files are deleted by calling  MPI_File_delete.

Now let us have a look at what the blah... blah... blah... stands for.

First every process seeds the random number generator differently and generates the integers:

      srand(28 + my_rank);
      for (i = 0; i < number_of_integers; i++) *(junk + i) = rand();

Then we execute a timed write on the open file using MPI_File_write and capture the error, if there is any to be caught. The error is treated similarly to the way we did it for MPI_File_open. We don't MPI_Abort if there is a problem. Instead we set my_write_error to TRUE and then flow with it.

      start = MPI_Wtime();
      my_write_error = 
        MPI_File_write(fh, junk, number_of_integers, MPI_INT, &status);
      if (my_write_error != MPI_SUCCESS) {
        MPI_Error_class(my_write_error, &error_class);
        MPI_Error_string(error_class, error_string, &length_of_error_string);
        printf("%3d: %s\n", my_rank, error_string);
        MPI_Error_string(my_write_error, error_string, &length_of_error_string);
        printf("%3d: %s\n", my_rank, error_string);
        my_write_error = TRUE;
      }
      else {
        finish = MPI_Wtime();
        io_time = finish - start;
        printf("%3d: io_time = %f\n", my_rank, io_time);
      }

Before we exit the if clause we still check if any of the processes has encountered writing problems and set the variable write_error accordingly.

      MPI_Allreduce (&my_write_error, &write_error, 1, MPI_INT,
                     MPI_LOR, MPI_COMM_WORLD);

#ifdef DEBUG
      if (i_am_the_master)
        if (write_error)
          fprintf(stderr, "problem writing on files\n");
#endif

At the end of all these adventures the processes meet at MPI_Finalize and exit.