Submitting POE/MPI Jobs

Here is an example of an MPI version of "hello world". The program itself looks as follows:

#include <stdio.h>
#include <mpi.h>

main(argc, argv)
int argc;
char *argv[];
{
        char name[BUFSIZ];
        int length;

        MPI_Init(&argc, &argv);
        MPI_Get_processor_name(name, &length);
        printf("%s: hello world\n", name);
        MPI_Finalize();
}

Compile it with the command

gustav@sp20:../LoadLeveler 17:07:04 !513 $ mpcc mpi-hello.c -o mpi-hello
gustav@sp20:../LoadLeveler 17:07:23 !514 $

and run by submitting the following LoadLeveler script:

gustav@sp20:../LoadLeveler 19:27:47 !647 $ cat mpi-hello.ll
# @ job_type = parallel
# @ environment = COPY_ALL; MP_EUILIB=ip; MP_INFOLEVEL=3
# @ requirements = (Adapter == "hps_ip")
# @ min_processors = 4
# @ max_processors = 8
# @ class = test
# @ notification = always
# @ executable = /usr/bin/poe
# @ arguments = mpi-hello
# @ output = mpi-hello.out
# @ error = mpi-hello.err
# @ queue
gustav@sp20:../LoadLeveler 19:27:49 !648 $ llsubmit mpi-hello.ll

This is what the file mpi-hello.out may look like after the run is finished:

gustav@sp20:../LoadLeveler 19:27:49 !648 $ cat mpi-hello.out
sp21.ucs.indiana.edu: hello world
sp19.ucs.indiana.edu: hello world
sp24.ucs.indiana.edu: hello world
sp20.ucs.indiana.edu: hello world
sp22.ucs.indiana.edu: hello world
sp23.ucs.indiana.edu: hello world
sp17.ucs.indiana.edu: hello world
sp18.ucs.indiana.edu: hello world
gustav@sp20:../LoadLeveler 19:29:11 !649 $

The file mpi-hello.err contains messages from poe:

gustav@sp20:../LoadLeveler 19:29:11 !649 $ cat mpi-hello.err
INFO: DEBUG_LEVEL changed from 0 to 1
D1<L1>: ./host.list file did not exist
D1<L1>: mp_euilib = ip
D1<L1>: node allocation strategy = 1
ATTENTION: 0031-408  8 nodes allocated by LoadLeveler, continuing...
INFO: 0031-119  Host sp21.ucs.indiana.edu allocated for task 0
INFO: 0031-119  Host sp24.ucs.indiana.edu allocated for task 1
INFO: 0031-119  Host sp19.ucs.indiana.edu allocated for task 2
INFO: 0031-119  Host sp22.ucs.indiana.edu allocated for task 3
INFO: 0031-119  Host sp20.ucs.indiana.edu allocated for task 4
INFO: 0031-119  Host sp23.ucs.indiana.edu allocated for task 5
INFO: 0031-119  Host sp17.ucs.indiana.edu allocated for task 6
INFO: 0031-119  Host sp18.ucs.indiana.edu allocated for task 7
...

Observe that since there was no file host.list in my working directory, LoadLeveler consulted the Resource Manager daemons and allocated the nodes for the run on its own. Only nodes from the group supporting the test class, in this case, would be selected.

If you need to perform certain manipulations before and after running the main executable you can use scripting the same way as has already been discussed for sequential programs. For example, you could run the MPI "hello world" example as follows:

gustav@sp20:../LoadLeveler 19:35:59 !668 $ cat mpi-hello-2.ll
# @ shell = /afs/ovpit.indiana.edu/@sys/gnu/bin/bash
# @ job_type = parallel
# @ environment = COPY_ALL; MP_EUILIB=ip; MP_INFOLEVEL=3
# @ requirements = (Adapter == "hps_ip")
# @ min_processors = 4
# @ max_processors = 8
# @ output = mpi-hello.out
# @ error = mpi-hello.err
# @ class = test
# @ notification = never
# @ queue
poe mpi-hello
gustav@sp20:../LoadLeveler 19:36:10 !669 $

Observe that I have instructed LoadLeveler not to send me any e-mail at all, regardless of what is going to happen to the job. You may wish to use #@notification=never if you submit a lot of short jobs to the test class.

The next example shows how you can have even more control over the way your POE job is run under LoadLeveler:

gustav@sp20:../LoadLeveler 19:41:24 !683 $ cat mpi-hello-3.ll
# @ shell = /afs/ovpit.indiana.edu/@sys/gnu/bin/bash
# @ job_type = parallel
# @ environment = COPY_ALL; 
# @ requirements = (Adapter == "hps_ip")
# @ min_processors = 4
# @ max_processors = 8
# @ output = mpi-hello.out
# @ error = mpi-hello.err
# @ class = test
# @ notification = never
# @ queue
> host.list.$LOADL_STEP_ID
NPROC=0
for node in $LOADL_PROCESSOR_LIST
do
   echo $node >> host.list.$LOADL_STEP_ID
   NPROC=`expr $NPROC + 1`
done
#
export MP_HOSTFILE=host.list.$LOADL_STEP_ID
export MP_PROCS=$NPROC
export MP_EUILIB=ip
export MP_EUIDEVICE=css0
export MP_INFOLEVEL=3
#
poe mpi-hello 
#
rm $MP_HOSTFILE
gustav@sp20:../LoadLeveler 19:41:28 !684 $

This time I construct the POE host file dynamically and pass it to POE via the MP_HOSTFILE environmental variable. At the same time I count the number of allocated nodes. I have requested that number to be between 4 and 8, but the exact number will be known only when the job starts. That number is passed to POE via the MP_PROCS environmental variable.

Here is what the mpi-hello.err file produced by the run looks like:

gustav@sp20:../LoadLeveler 19:43:58 !685 $ cat mpi-hello.err
INFO: DEBUG_LEVEL changed from 0 to 1
D1<L1>: Open of file host.list.sp20.ucs.indiana.edu.58.0 successful
D1<L1>: mp_euilib = ip
D1<L1>: node allocation strategy = 1
INFO: 0031-119  Host sp18.ucs.indiana.edu allocated for task 0
INFO: 0031-119  Host sp22.ucs.indiana.edu allocated for task 1
INFO: 0031-119  Host sp19.ucs.indiana.edu allocated for task 2
INFO: 0031-119  Host sp21.ucs.indiana.edu allocated for task 3
INFO: 0031-119  Host sp17.ucs.indiana.edu allocated for task 4
INFO: 0031-119  Host sp23.ucs.indiana.edu allocated for task 5
INFO: 0031-119  Host sp20.ucs.indiana.edu allocated for task 6
INFO: 0031-119  Host sp24.ucs.indiana.edu allocated for task 7
...

The listing shows that host names were obtained from the dynamically generated host file, and it also shows which task runs on which host.