About PESSL

Next: About Secure Shell Up: Spectral Methods Previous: A Comment about the

About PESSL

PESSL stands for Parallel Engineering and Scientific Subroutine Library.
PESSL is the basic research tool on the IBM SP multicomputer.
PESSL lives in /usr/lpp/pessl on all SP nodes.

PESSL documentation lives on

/usr/lpp/pessl/doc/postscript/pessl.00.ps

To view this file configure your environment so that the following directories are (preferably) in front of your command search PATH:

/afs/ovpit.indiana.edu/@sys/gnu/bin
/afs/ovpit.indiana.edu/@sys/X11R6.4/bin
/afs/ovpit.indiana.edu/@sys/teTeX/bin          
/afs/ovpit.indiana.edu/@sys/HDF/bin
/afs/ovpit.indiana.edu/@sys/netpbm/bin

You should also configure MANPATH in the same order:

/afs/ovpit.indiana.edu/@sys/gnu/man
/afs/ovpit.indiana.edu/@sys/X11R6.4/man
/afs/ovpit.indiana.edu/@sys/teTeX/man
/afs/ovpit.indiana.edu/@sys/HDF/man
/afs/ovpit.indiana.edu/@sys/netpbm/man

Having done all that and having sourced the new environment type:

$ ghostview /usr/lpp/pessl/doc/postscript/pessl.00.ps

As a reminder: in order to get access to AFS directories that are ACL controlled, you must authenticate yourself to AFS. Configure the following in your command search PATH:
```
/usr/afsws/bin
```
and then add
```
/usr/afsws/man
```
to your MANPATH too. Use the command
```
$ klog `whoami`
```
to obtain AFS tokens. Use the command
```
$ tokens
```
to inspect the state of your AFS tokens. They will expire automatically after 25 hours. To destroy your AFS tokens issue the command:
```
$ unlog
```
Normally you should unlog explicitly from the AFS before disconnecting from your UNIX or NT session - that is, unless you want the tokens to live for, say, background jobs. If you have problems remembering your AFS password, use the command:
```
$ kpasswd
```
to change it. You can change your OVPIT AFS password from any machine (around the world, sic!) that mounts AFS.
The diffusion example we've been talking about can be found on pages APC-5 through APC-8, but, beware, it is not explained in great detail, and there is an error in one place (the authors forgot about the change of sign when integrating by parts).
The HPF example code is discussed on pages APC-41 through APC-61. This code is not very neatly written, that is, it is too F77 in style. We will attempt to

1.
Make it more F90;
2.
Add HDF animation to it;
3.
Annotate it.
The code itself lives in
```
/usr/lpp/pessl/example/hpf
```

To compile the code do as follows:

gustav@sp20:../SP 16:33:45 !527 $ cd /usr/lpp/pessl/example
gustav@sp20:../example 16:33:51 !528 $ cp -r hpf ~/diffusion
gustav@sp20:../example 16:33:59 !529 $ cd ~/diffusion
gustav@sp20:../diffusion 16:34:05 !530 $ make
xlhpf -c -qreport=hpflist -C -O param.f
** parameters   === End of Compilation 1 ===
1501-510  Compilation successful for file param.f.
xlhpf -c -qreport=hpflist -C -O diffusion.f
** diffusion   === End of Compilation 1 ===
1501-510  Compilation successful for file diffusion.f.
xlhpf -c -qreport=hpflist -C -O fourier.f
** fourier   === End of Compilation 1 ===
1501-510  Compilation successful for file fourier.f.
xlhpf -c -qreport=hpflist -C -O main.f
** main   === End of Compilation 1 ===
1501-510  Compilation successful for file main.f.
xlhpf -o diffusion main.o param.o diffusion.o fourier.o  \
      -lblacs -lpessl -lpesslhpf -lessl 
gustav@sp20:../diffusion 16:34:36 !531 $

Although it is possible to make the SP understand AFS entirely, so that AFS version of rsh, which lives in /usr/afsws/bin is used to make AFS-authenticated connections to other nodes when running a parallel job, we don't have things set up that way on our system. For this reason, it's best to place your binaries and data file on a file system that is local to the SP before a production run. It is also much more efficient to do it this way, because access to data and binaries on any remote server is always very slow, compared to a local access.
The libraries linked with the program are:
/usr/lib/libblacs.a
Basic Linear Algebra Communication Subprograms, BLACS: this is an on-going project
[...] whose purpose is to create a linear algebra oriented message passing interface that may be implemented efficiently and uniformly across a large range of distributed memory platforms.
The main investigator of the project is Jack Dongarra, and the person who does most of the work is R. Clint Whaley. Both are from the University of Tennessee. For more information about BLACS see:
```
http://www.netlib.org/blacs/
```
/usr/lib/libpessl.a
Parallel Engineering and Scientific Subroutine Library: this is not a stand-alone library. It depends on BLACS and on ESSL.
/usr/lib/libpesslhpf.a
This is an auxiliary library that provides interface to HPF programs. PESSL works both with MPI and with HPF codes.
/usr/lib/libessl.a
Engineering and Scientific Subroutine Library. This is a sequential library that PESSL is built upon. It is based, amongst other things, on BLAS, i.e., Basic Linear Algebra Subprograms. These subprograms derive from Jack Dongarra's work too, and are available from Netlib (http://www.netlib.org). For more information about BLAS see:
```
http://www.netlib.org/blas/
```

There is a script called run.script provided with the example, but it won't work on our system, because of the way things are set up (e.g., you can use the User-Space mode from LoadLeveler jobs only).

First you need to create a .rhosts file in your home directory. That file should look as follows:
```
gustav@sp20:../SP 17:04:09 !576 $ cat .rhosts
sp17
sp18
sp19
sp20
sp21
sp22
sp23
sp24
sp40
sp41
sp42
sp43
gustav@sp20:../SP 17:04:23 !577 $
```
These are all the nodes that we are allowed to work with in this course. You can use them for interactive runs, or you can submit your jobs to either the pa class or to the test class.
You should also copy .rhosts or link to a file called host.list in your run directory, i.e., the directory where you've got your binaries and your data files:
```
gustav@sp20:../SP 17:04:23 !578 $ cd diffusion
gustav@sp20:../SP 17:04:23 !579 $ ln -s ~/.rhosts host.list
gustav@sp20:../SP 17:04:23 !580 $
```

Now you can run the program as follows:

gustav@sp20:../SP 17:04:23 !580 $ poe diffusion -euidevice css0 \
                                    -euilib ip -procs 8 -ilevel 3 \
                                    > diffusion.log 2> diffusion.err
gustav@sp20:../SP 17:04:23 !581 $

This is a very short program and therefore it can be run interactively. Real production jobs must be submitted to LoadLeveler.
After the run has completed, you should see its output on diffusion.log and various diagnostic messages and/or errors on diffusion.err.

The beginning of diffusion.err looks as follows:

INFO: DEBUG_LEVEL changed from 0 to 1
D1<L1>: Open of file ./host.list successful
D1<L1>: mp_euilib = ip
D1<L1>: task 0 sp17.ucs.indiana.edu 129.79.7.68 10
D1<L1>: task 1 sp18.ucs.indiana.edu 129.79.7.69 10
D1<L1>: task 2 sp19.ucs.indiana.edu 129.79.7.70 10
D1<L1>: task 3 sp20.ucs.indiana.edu 129.79.7.71 10
D1<L1>: task 4 sp21.ucs.indiana.edu 129.79.7.72 10
D1<L1>: task 5 sp22.ucs.indiana.edu 129.79.7.73 10
D1<L1>: task 6 sp23.ucs.indiana.edu 129.79.7.74 10
D1<L1>: task 7 sp24.ucs.indiana.edu 129.79.7.75 10

This means that this job has run successfully on 8 SP nodes, sp17 through sp24.

You will not be provided with this information, unless you request -ilevel 3 or higher. To request -ilevel other than the default, which is 1, or 0 is costly and should not be normally done for production runs.

The file diffusion.log yields the normal output of the program, i.e., tabulated diffusion results for various points and times. Our first task will be to replace that with an HDF animation.

This is what the beginning of that file looks like:

    point #      X         Y
       1        .5236      .5236
       2       1.0472      .5236
       3       1.5708      .5236
       4       2.0944      .5236
       5       2.6180      .5236
    ...
      21        .5236     2.6180
      22       1.0472     2.6180
      23       1.5708     2.6180
      24       2.0944     2.6180
      25       2.6180     2.6180
 
 
                              Points
   TIME        #   1     #   2     #   3     #   4     #   5     #   6
    .10000   1.87754   2.99610   3.37190   2.99610   1.87754   3.02958
    .20000   1.61896   2.62389   2.93406   2.55544   1.51137   2.63140
    .30000   1.40707   2.29776   2.55618   2.18506   1.25438   2.30026
    .40000   1.22871   2.01193   2.22646   1.87602   1.05857   2.01355
    .50000   1.07478   1.75988   1.93729   1.61515    .90145   1.76125
   ...
   1.50000    .26483    .42718    .45803    .36980    .20098    .42757
   1.60000    .22898    .36914    .39549    .31908    .17332    .36947
   1.70000    .19790    .31888    .34143    .27530    .14948    .31917
   1.80000    .17098    .27539    .29472    .23753    .12893    .27564
   1.90000    .14767    .23778    .25437    .20493    .11121    .23800
   2.00000    .12752    .20527    .21952    .17680    .09592    .20546

Next: About Secure Shell Up: Spectral Methods Previous: A Comment about the

Zdzislaw Meglicki
2001-02-26