Some General Comments about the SP Environment

• The nodes we can use for this course all have   AFS mounted. This means that you can access your files on the AFS from the SP, and you can also access the usual set of AFS applications that we've been working with on the Ships cluster.
• The AFS applications can be found, as usual, in

  /afs/ovpit.indiana.edu/@sys
  

  where @sys this time evaluates to rs_aix42.
• There are five major directories there at present:

  HDF

  The NCSA Hierarchical Data Format libraries and utilities.  

  X11R6.4

  The latest version of X-windows from the X consortium.  

  gnu

  As much GNU software as I managed to get going over the Thanksgiving Recess.  

  netpbm

  The PBM+ (and more) library of   utilities for manipulating images.

  teTeX

  TEX directory, which also includes FWEB and L^ATEX2HTML. 

  There's no Collage in the HDF directory at present, but you can always run Collage on the SGI machines and view data generated on the SP and deposited on the AFS. That process should be quite transparent.
• AIX compilers are called  

  xlc

  for C

  xlC

  for C++

  xlf90

  for F90

  xlhpf

  for HPF

  They are very strictly adherent to ANSI standards where such exist. Only a subset HPF is implemented, but it's quite usable.
• AIX Fortran insist on suffix .f and rejects files with suffix .f90. Here's a GNU make based remedy to that problem:  

  OS             = $(shell uname -s)
  ...
  ifeq ($(OS), AIX)
  %.o: %.f90
          ln -s $< `basename $< .f90`.f
          $(F90) $(F90_OPTS) -c $(F90_MODULES) `basename $< .f90`.f
          rm `basename $< .f90`.f
  else
  %.o: %.f90
          $(F90) $(F90_OPTS) -c $(F90_MODULES) $<
  endif
  

• The location of module directories is indicated with the -I switch, as is the case also under IRIX.
• The .o files generated by compiling modules must be included in the linking step, even if the modules contained declarations only. That wasn't necessary under IRIX and Solaris. I have generated an additional library in the HDF area, which takes care of that. The library is libhdf-modules.a and you should link it in when generating HDF programs.
• Here is the example of our HDF Makefile with additional customisation for AIX:

  HDF_LIB        = /afs/ovpit.indiana.edu/@sys/HDF/lib
  HDF_MODULES    = /afs/ovpit.indiana.edu/@sys/HDF/modules
  OS             = $(shell uname -s)
  RM             = /bin/rm
  HDF_EXAMPLES   = hdf-1 hdf-2 hdf-3 hdf-4 hdf-5 hdf-6 hdf-7 hdf-8
  
  ifeq ($(OS), IRIX)
     F90         = f90
     F90_OPTS    = -n32 -g
     LIBS        = -lmfhdf -ldf -ljpeg -lz -lm
     F90_MODULES = -I$(HDF_MODULES)
  else
     ifeq ($(OS), SunOS)
        F90         = f90
        F90_OPTS    = -g
        LIBS        = -lmfhdf -lnsl -ldf -ljpeg -lz -lm
        F90_MODULES = -M$(HDF_MODULES)
     else
        ifeq ($(OS), AIX)
           F90         = xlf90
           F90_OPTS    = -g
           LIBS        = -lhdf-modules -lmfhdf -ldf -ljpeg -lz -lm
           F90_MODULES = -I$(HDF_MODULES)
        else
           F90_OPTS    = -g
           LIBS        = -lmfhdf -ldf -ljpeg -lz -lm
           F90_MODULES = -I$(HDF_MODULES)
        endif
     endif
  endif
  
  ifeq ($(OS), AIX)
  %.o: %.f90
          ln -s $< `basename $< .f90`.f
          $(F90) $(F90_OPTS) -c $(F90_MODULES) `basename $< .f90`.f
          rm `basename $< .f90`.f
  else
  %.o: %.f90
          $(F90) $(F90_OPTS) -c $(F90_MODULES) $<
  endif
  
  %: %.o
          $(F90) $(F90_OPTS) -o $@ $< -L$(HDF_LIB) $(LIBS)
  
  all: $(HDF_EXAMPLES)
  
  clean:
          $(RM) -f *.o core
  
  clobber: clean
          $(RM) -f $(HDF_EXAMPLES)
  

• And here is how this file works under AIX:

  gustav@sp20:../src 21:32:13 !547 $ make clobber
  /bin/rm -f *.o core
  /bin/rm -f hdf-1 hdf-2 hdf-3 hdf-4 hdf-5 hdf-6 hdf-7 hdf-8
  gustav@sp20:../src 21:32:18 !548 $ make hdf-1
  ln -s hdf-1.f90 `basename hdf-1.f90 .f90`.f
  xlf90 -g -c -I/afs/ovpit.indiana.edu/@sys/HDF/modules \
     `basename hdf-1.f90 .f90`.f
  ** create_sds   === End of Compilation 1 ===
  1501-510  Compilation successful for file hdf-1.f.
  mv: 0653-404 hdf-1.o: Unable to duplicate owner and mode after move.
  rm `basename hdf-1.f90 .f90`.f
  xlf90 -g -o hdf-1 hdf-1.o -L/afs/ovpit.indiana.edu/@sys/HDF/lib \
     -lhdf-modules -lmfhdf -ldf -ljpeg -lz -lm
  rm hdf-1.o
  gustav@sp20:../src 21:32:24 !549 $
  

  The message about Unable to duplicate$\ldots$ pops up on the AFS only. It's not serious.
• There's rather little space available for users on the SP at present. Your home directories are served via NFS from an HP server. This is very inefficient - you should avoid doing large I/O transactions with your $HOME or your AFS directory from production runs. Use local scratch space in

  /scr
  

  which is writable to everybody. Transfer final results only to your NFS or AFS directory.  
• Note, however, that /scr is local to each node, and so cannot be used for parallel jobs transparently. You must write on it explicitly from an MPI program.
• No need to worry about any of that too much for our class-room exercises and examples. NFS and AFS will suffice.
• If you want   to run parallel jobs on the AFS you must ensure that your AFS working directory is made writable to the world, because our SP has not been configured to transfer AFS tickets between the nodes. Issue the command:

  $ fs setacl . -acl system:anyuser rlidw
  

  while in that directory. To protect the directory from prying eyes of unauthorised users you can always close it off with the command:

  $ fs setacl . -acl system:anyuser none
  

  after the run has finished, or you can bury it deeply inside your AFS directory tree, and close directories above it.

  You should also check that all nodes specified in your host.list file run AFS. Sometimes they don't and that may result in run-time errors.

• The SP also provides a General Parallel File System,     GPFS, but it hasn't been configured on our machine yet. GPFS yields the most efficient mechanism for large I/O transactions from production jobs. Do not use GPFS (once it's been configured) for long term storage though, even though there's going to be a lot of contiguous disk space there.