Nonblocking Communications

Non-blocking communications, send and receive, have a number of advantages, e.g., they won't hang a program and they can be interleaved with useful work, e.g., computation. But they are harder to use.

A non-blocking send command really means a send start. The function call returns as soon as possible, i.e., as soon as other parts of the operating system and/or hardware can take over. But the send process itself may not be complete still for a long time. A separate send complete call is necessary to verify that the data has been copied out of the send buffer and that the buffer can now be used for another communication.

The same applies to a non-blocking receive. The operation is initialised with a receive start, and then, if the operating system and hardware allow for that, it continues in the background, until a separate receive complete call is issued, whose purpose is to verify that all the data has been received in the receive buffer.

Non-blocking send starts can be used with the same four modes as blocking sends, i.e., standard, buffered, synchronous, and ready.

Non-blocking sends can be matched with blocking receives and vice versa.

The synopsis for the non-blocking standard send is as follows. In C:

int MPI_Isend(void* buf, int count, MPI_Datatype datatype, int dest,
              int tag, MPI_Comm, comm, MPI_Request *request)

In Fortran:

mpi_isend(buf, count, datatype, dest, tag, comm, request, ierror)
   <type> buf(*)
   integer count, datatype, dest, tag, comm, request, ierror

and likewise for MPI_Ibsend, MPI_Issend, and MPI_Irsend.

The non-blocking call, compared to the blocking one, needs one more parameter: MPI_Request *request. This is a so called opaque object, which identifies communication operations and matches the operation that initiates the communication with the operation that terminates it.

How do you terminate a non-blocking send?

The easiest way is to issue in C

int MPI_Wait(MPI_Request *request, MPI_Status *status)

or, in Fortran:

mpi_wait(request, status, ierror)
   integer request, status(mpi_status_size), ierror

This call will make your process hang until the operation identified by the request is complete. The call to MPI_Wait deallocates the request and sets the request handle to MPI_REQUEST_NULL.

To follow MPI_Isend immediately with MPI_Wait is the same as to call MPI_Send. But splitting the latter into the former lets you do a number of other things between the calls to MPI_Isend and MPI_Wait.

The outcome of the operation is returned in status, which may be queried using MPI_Test_cancelled.

The other way to complete a non-blocking send is to issue in C

int MPI_Test(MPI_Request *request, int *flag, MPI_Status *status)

or in Fortran

mpi_test(request, flag, status, ierror)
   logical flag
   integer request, status(MPI_STATUS_SIZE), ierror

Unlike MPI_Wait this call doesn't hang waiting for the communication request to get completed. It returns right away with flag = true if the operation is complete and the value of request is set to MPI_REQUEST_NULL. Otherwise flag = false and the value of request remains unchanged. Most commonly you are likely to use MPI_Test in a loop: checking if the communication has completed, then doing something else, then checking again, and so on.

The returned status can be tested with a call to MPI_Test_cancelled.

The non-blocking equivalent of MPI_Recv is in C:

int MPI_Irecv(void* buf, int count, MPI_Datatype datatype, int source,
              int tag, MPI_Comm comm, MPI_Request, *request)

or in Fortran:

mpi_irecv(buf, count, datatype, source, tag, comm, request, ierror)
   <type> buf(*)
   integer count, datatype, source, tag, comm, request, ierror

Observe that there is no slot for status in this call. The status may not be ready yet. You will have to use either MPI_Wait or MPI_Test to obtain the status.

MPI_Irecv followed immediately by MPI_Wait is equivalent to MPI_Recv. But, as was the case with MPI_Send, having split MPI_Recv into MPI_Irecv and MPI_Wait lets you do some other work in between the two, thus masking the communication.

If you are at the receiving end there are additional calls you can issue in order to probe an incoming message without receiving it. The operations are MPI_Iprobe, MPI_Probe, and MPI_Cancel.

The operation MPI_Iprobe has the following synopsis in C:

int MPI_Iprobe(int source, int tag, MPI_Comm comm, int *flag,
               MPI_Status *status)

and in Fortran:

mpi_iprobe(source, tag, comm, flag, status, ierror)
   logical flag
   integer source, tag, comm, status(mpi_status_size), ierror

It is a non-blocking operation. It returns flag = true if there is a message that can be received and that matches source, tag, and comm. The status associated with the message is returned in status and can be further inspected for the length of the message, type of data, etc. If flag = false then there is no message, and nothing worth looking at is returned in status. The subsequent MPI_Recv will receive the message identified by the probe assuming that no other thread has snatched the message in the meantime.

The blocking counterpart of MPI_Probe hangs on until a matching message has been found. There is no flag argument there. In C:

int MPI_Probe(int source, int tag, MPI_Comm comm, MPI_Status *status)

and in Fortran:

mpi_probe(source, tag, comm, status, ierror)
   integer source, tag, comm, status(MPI_STATUS_SIZE), ierror

A non-blocking send or a receive can be cancelled, i.e., discarded with, in C:

int MPI_Cancel(MPI_Request *request)

or, in Fortran:

mpi_cancel(request, ierror)
   integer request, ierror

This call marks the request for cancellation, but the communication itself still doesn't complete, until you issue, in C:

int MPI_Request_free(MPI_Request *request)

or, in Fortran:

mpi_request_free(request, ierror)
   integer request, ierror

You can also complete the communication with MPI_Wait or MPI_Test, but since you're not interested in the message there is little point waiting or testing for it.

Now, how do you examine status once you have it?

Status is a structure in C and an array in Fortran with multiple fields. Three fields compulsory and self-explanatory: status.MPI_SOURCE, status.MPI_TAG, and status.MPI_ERROR. In Fortran these are: status(MPI_SOURCE), status(MPI_TAG), and status(MPI_ERROR). But status contains also additional information, which is not directly accessible. There is a function call MPI_Get_count, which you can use to inquire about the length of the message, for example, before you're going to receive it.

The synopsis of MPI_Get_count is, in C:

int MPI_Get_count(MPI_Status *status, MPI_Datatype datatype, int *count)

and in Fortran

mpi_get_count(status, datatype, count, ierror)
   integer status(MPI_STATUS_SIZE), datatype, count, ierror

Here is an example code, which illustrates the use of some of the non-blocking communication functions discussed so far.

#include <stdio.h>
#include <mpi.h>

main(argc, argv)
int argc;
char *argv[];
{
  
  int pool_size, my_rank;

  MPI_Init(&argc, &argv);
  MPI_Comm_size(MPI_COMM_WORLD, &pool_size);
  MPI_Comm_rank(MPI_COMM_WORLD, &my_rank);
  
  if (my_rank == 0) {

    char send_buffer[BUFSIZ], my_cpu_name[BUFSIZ];
    int my_name_length;
    MPI_Request request;
    MPI_Status status;

    MPI_Get_processor_name(my_cpu_name, &my_name_length);
    sprintf (send_buffer, "Dear Task 1,\n\
Please do not send any more messages.\n\
Please send money instead.\n\
\tYours faithfully,\n\
\tTask 0\n\
\tRunning on %s\n", my_cpu_name);
    MPI_Isend (send_buffer, strlen(send_buffer) + 1, MPI_CHAR,
               1, 77, MPI_COMM_WORLD, &request);
    printf("hello there user, I've just started this send\n\
and I'm having a good time relaxing.\n");
    MPI_Wait (&request, &status);
    printf("hello there user, it looks like the message has been sent.\n");

    if (request == MPI_REQUEST_NULL) {
      printf("\tthe send request is MPI_REQUEST_NULL now\n");
    } else {
      printf("\tthe send request still lingers\n");
    }

  } 
  else if (my_rank == 1) {

    char recv_buffer[BUFSIZ], my_cpu_name[BUFSIZ];
    int my_name_length, count;
    MPI_Request request;
    MPI_Status status;
    
    MPI_Get_processor_name(my_cpu_name, &my_name_length);
    MPI_Irecv (recv_buffer, BUFSIZ, MPI_CHAR, 0, 77, MPI_COMM_WORLD,
               &request);
    printf("hello there user, I've just started this receive\n\
on %s, and I'm having a good time relaxing.\n", my_cpu_name);
    MPI_Wait (&request, &status);
    MPI_Get_count (&status, MPI_CHAR, &count);
    printf("hello there user, it looks like %d characters \
have just arrived:\n", count );
    printf("%s", recv_buffer);

    if (request == MPI_REQUEST_NULL) {
      printf("\tthe receive request is MPI_REQUEST_NULL now\n");
    } else {
      printf("\tthe receive request still lingers\n");
    }

  }

  MPI_Finalize();

}

And here is how this program compiles and runs:

gustav@sp20:../MPI 18:35:56 !553 $ mpcc -g -o non-block non-block.c
gustav@sp20:../MPI 18:36:45 !554 $ non-block -procs 2 -labelio yes
  1:hello there user, I've just started this receive
  1:on sp19.ucs.indiana.edu, and I'm having a good time relaxing.
  0:hello there user, I've just started this send
  0:and I'm having a good time relaxing.
  0:hello there user, it looks like the message has been sent.
  0:    the send request is MPI_REQUEST_NULL now
  1:hello there user, it looks like 139 characters have just arrived:
  1:Dear Task 1,
  1:Please do not send any more messages.
  1:Please send money instead.
  1:    Yours faithfully,
  1:    Task 0
  1:    Running on sp17.ucs.indiana.edu
  1:    the receive request is MPI_REQUEST_NULL now
gustav@sp20:../MPI 18:36:51 !555 $