Center for Fluid Mechanics, Turbulence and Computation, Brown University

Outline:

SP2 Hardware/Configuration

Nodes
- 24 IBM Power2 "Thin 2" Nodes
  - 66.7 MHz, 267 MFlop/s peak, >230 MFlop/s in matrix multiplies
  - 128 KB primary data cache (4-way set associative, write-back, 128 byte lines)
  - 32 KB primary instruction cache (2-way set associative, 128 byte lines)
  - 128 bit data path to main memory, 256 bit between data cache and registers
  - Close to "Wide Node" performance in "Thin Node" size
- 128 MB RAM per node, about 90-100 MB available for user programs

"Omega" or "Butterfly" flat network topology

IBM High Performance Switch performance measurements

 Protocol        Node Type       Latency       Pt to Pt Bandwidth
 --------------  -------------   ------------  -----------
 MPL IP          66 MHz Thin      312.1 usec    9.9 MB/Sec.
                 66 MHz Thin-2    270.4 usec   12.0 MB/Sec.
                 66 MHz Wide      268.8 usec   12.1 MB/Sec.
 MPL user space  66 MHz Thin       40.0 usec   35.4 MB/Sec.
                 66 MHz Thin-2     39.0 usec   35.7 MB/Sec.
                 66 MHz Wide       39.2 usec   35.6 MB/Sec.
 MPI IP          66 MHz Wide     <300   usec   >12  MB/Sec.(preliminary 
                                                            results)
 MPI user space  66 MHz Wide      <45   usec   >35  MB/Sec.(preliminary
                                                           results)

ie.

35MB/s achievable "ping-pong" (send-receive-send) bandwidth
40 microsecond latency with IBM software (52 microsecond with MPICH)

Ethernet also connects the nodes

Setup
- Control Workstation
- IBM RS/6000 550 - zeus
- Used for job management only (no compiling anymore)

IBM RS/6000 AIX Programming Environment

AIX - IBM's version of Unix. Currently running version 4.1.4 but will be upgrading to 4.3. All usual Unix commands work, but AIX is built using mixed UNIX System V and Berkeley Software Distribution 4.3 (4.3 BSD) conventions (without actually using any ATT or BSD code), with later conformance to the Portable Operating System Interface for Computer Environments (POSIX) IEEE 1003.1-1990; users familiar with SunOS 4.1.* or Ultrix should better check the man pages for some commands
Compilers
- C compiler: cc (also callable as xlc, c89 - for the ANSI conforming language level version)
- Fortran compiler: f77 (also callable as xlf and xlf90). This compiler handles both Fortran 77 and Fortran 90. Fortran 90 files must have a .f file extension instead of the more common .f90 one.
- IBM XL HPF compiler: xlhpf. This is a subset HPF compiler, highly optimized for the SP2.
- C++ compiler: xlC
Optimizing preprocessors (Kuck & Associates, Inc):
- C preproccesor - kapc (kxlc & kcc drivers)
- F77 preprocessor - kapf
Numerical libraries:
- ESSL - IBM's optimized numerical subroutine library - best performance for BLAS, FFTs etc. Using it should provide close to the best single processor performance possible.
- LAPACK - From Netlib - uses ESSL BLAS routines and equivalent ESSL routines where possible (use -llapack_esslp2.a instead of -llapackp2).
Debuggers:
- dbx
- xldb
Profiling tools:
- prof
- gprof
New ever since the AIX 4.1.4 upgrade:
- LPEX programmable editor for Fortran, C and C++
- Program Builder makefile generation tool

IBM RS/6000 SP Parallel Environment

POE (Parallel Operating Environment)
Parallel Compilers
- Currently only the IBM HPF compiler
- Given the immaturity of HPF compilers performance is expected to vary from really low to quite decent for problems that are very data-parallel, with IBM's compiler probably offering the best performance.
Other parallel compilers (eg. Split-C) could be installed if needed (contact ce107@cfm.brown.edu)

PESSL Parallel version of ESSL for parallel BLAS, FFTs and some ScaLAPACK routines. Requires use of the BLACS.

Message Passing Libraries:

MPI (Message Passing Interface) (This tutorial will deal exclusively with MPI)

The emerging Message Passing Standard: Currently at version 1.1, it supports more functionality than most if not all other alternatives. Recommended for all future portable code development. Disadvantages include the inability to spawn processes (as in PVM) (the Ohio Supercomputing Center's implementation of MPI called LAM corrects that) and that it is (currently) a less robust environment on workstations compared to PVM. The trend is towards use of MPI instead of PVM though. Also missing from the MPI standard is parallel I/O. Some of these problems will be resolved by the emerging MPI-2 standard. The SP2 at CFM supports:

MPICH - the Argonne portable MPI implementation. On the SP2, MPICH is built on top of IBM's native message passing library MPL and offers very good performance for point-to-point routines at least.
IBM's MPI - fully native IBM MPI to come with the AIX 4.1.4 upgrade to Parallel Environment 2.0. Currently available for AIX 3.2.5 only as a development version at National Supercomputer Centers. It provides the best performance on the SP2 platform, on par with that of MPL.

MPL (Message Passing Library)

Previously known as EUI and EUI-H this is IBM's initial high performance message passing library for the SP. It is more "lightweight" than MPI but offers less functionality and roughly the same performance as IBM's MPI. Currently will give the best performance on the machine but the drawback is that the code is non-portable. IBM is basing its future work on MPI mainly.

PVM (Parallel Virtual Machine)

Probably the best known of message-passing libraries PVM is a very robust by now system for performing parallel computations on heterogeneous networks of workstations and provides for fault-tolerant operation with dynamic process spawning making it useful for industrial applications. However on dedicated MPPs as the IBM SP2 its performance is not optimal and compared to MPI its functionality is more restrictive. The SP2 at CFM supports:

PVMe - IBM's proprietary modified PVM, offers better performance at slightly reduced functionality: it doesn't support dynamic addition/deletion of hosts. The current version is based on ORNL PVM 3.2.6 and will not work properly with codes using the extra functions that appeared with PVM 3.3. The version to come with AIX 4.1.4 is PVM 3.3 based.
ORNL PVM 3.3.10. Currenlty it offers much worse performance than PVMe but after the upgrade to AIX 4.1.4 and IBM's MPI, PVM 3.3.10 built on top of MPI will provide better performance than PVMe in terms of both latency and bandwidth.

NXLib (for Paragon users)

For people with Paragon codes: a quick and easy way to run their codes on the SP (or a network of workstations). Communications will be slow (as NXLib will be treating the SP2 as a collection of IBM workstations connected via ethernet), so moving on to MPI is encouraged.

Other message passing libraries

TCGMSG, P4 etc. could be installed if needed (contact ce107@cfm.brown.edu)

Parallel debuggers:

pdbx: IBM's parallel version of dbx
xpdbx: Motiff based version of pdbx

Visualization and Performance Monitoring Tool (VT): Trace Visualization for MPL codes and Performance Monitoring.

Program Marker Array (easy way to check program progress and load balancing). Instead of print statements used to monitor the code, one can use pmarray.

System Status Display (check the utilization of SP2 nodes)

Basic Parallel Programming

Attacking a problem with a parallel approach:

Data Parallel
Each processor does the same operations on a different subset of the data. In a distributed memory environment (such as the SP2) this is usually implemented in the SPMD - Single Program Multiple Data paradigm. In a shared memory environment, it can also be implemented using threads or tasks (in the Cray auto/microtasking fashion).
Work Parallel (also called Functional Parallel)
Each processor does different operations on part or all of the data, or sometimes even completely different data. This is usually implemented in the MPMD - Multiple Programs Multiple Data fashion, (the most common case of it being Master-Worker.
Domain Decomposition (the natural method of partitioning FE grids etc.)
Example: Calculating PI. Serial C code and MPI C code.

MPI Basics (Implementing the PI program)

More MPI routines

Point to Point Communication Routines
Collective Communication Routines
Datatypes in general and predefined datatypes
Communicators & Topologies

Examples from Linear Algebra

2-D Parallel FFT

Compiling, Linking and Running MPI applications

Compiler & linker drivers, include and library directories
- mpich_cc & mpich_xlf
- /usr/local/mpich/include
- /usr/local/mpich/lib/rs6000/ch_eui
Optimization switches (also see the xxlf command line builder

Basic optimization
-O -qarch=pwr2 -qtune=pwr2
High optimization
-O3 -qarch=pwr2 -qtune=pwr2
Optional high optimization flags:
- -qstrict (to avoid numerically dangerous transformations that change the order of operations - use it if you get wrong results)
- -qhot (hot spot transformations - the Fortran compiler will attempt to speed them up by reorder, unrolling etc.) - useful with cache information
- -Q (inlining)
- -qipa (Fortran interprocedural analysis - usually time consuming and currently does not work on the SP2)
Cache information (used by the Fortran compiler's -qhot option):
- -qcache=type=c:level=1:assoc=4:line=128:size=128 (data cache)
- -qcache=type=i:level=1:assoc=2:line=128:size=32 (instruction cache)
- -qcache:type=d:level=2:size=2048:assoc=2:line=4096 (TLB for data)
- -qcache:type=i:level=2:size=512:assoc=2:line=4096 (TLB for instructions)
Miscellaneous:
- -inlglue (Generate fast external linkage by inlining the code - pointer glue code - necessary at calls via a function pointer and calls to external procedures for C code)
KAP preprocessor flags:
- for C: +Kargs=-l:-r=3:-inl:-chs=128
- for Fortran: -Pk -Wp,-r=3,-inl,-f,-chs=128
Numerical Libraries (added at the end of the compile/link line)
- ESSL: -lesslp2
- LAPACK: -L/usr/local/lib -llapackp2 -lesslp2 (the latter is needed as LAPACK calls routines from ESSL
- BLACS: -lblacsp2
- PESSL: -lpesslp2 -lblacsp2 -lesslp2 (the last two are needed as PESSL calls routines from BLACS and ESSL)
Examples:
- C:
  
  Basic compilation & linking
  mpich_cc foo.c -o foo
  Basic optimizations
  mpich_cc -O -qarch=pwr2 -qtune=pwr2 foo.c -o foo
  Safe high optimizations
  mpich_cc -O3 -qstrict -qarch=pwr2 -qtune=pwr2 foo.c -o foo
  High optimizations
  mpich_cc -O3 -qarch=pwr2 -qtune=pwr2 foo.c -o foo
  Maximum optimizations recipe using KAP preprocessor
  mpich_cc -O3 -qinlglue -Q -qarch=pwr2 -qtune=pwr2 +Kargs=-l:-r=3:-inl:-chs=128 foo.c -o foo
- Fortran:
  
  Basic compilation & linking
  mpich_xlf foo.f -o foo
  Basic optimizations
  mpich_xlf -O -qarch=pwr2 -qtune=pwr2 foo.f -o foo
  Safe high optimizations
  mpich_xlf -O3 -qstrict -qarch=pwr2 -qtune=pwr2 foo.f -o foo
  High optimizations
  mpich_xlf -O3 -qarch=pwr2 -qtune=pwr2 foo.f -o foo
  Maximum optimizations recipe using compiler
  mpich_xlf -O3 -Q -qarch=pwr2 -qtune=pwr2 -qhot -qcache=type=c:level=1:assoc=4:line=128:size=128 -qcache=type=i:level=1:assoc=2:line=128:size=32 -qcache:type=d:level=2:size=2048:assoc=2:line=4096 -qcache:type=i:level=2:size=512:assoc=2:line=4096 foo.c -o foo
  Maximum optimizations recipe using KAP preprocessor
  mpich_xlf -O3 -Q -qarch=pwr2 -qtune=pwr2 -Pk -Wp,-r=3,-inl,-f,-chs=128 foo.f -o foo
- Porting Cray Fortran codes

Running

Node Pools
- Current Setup
  - Pool 0 (2 node interactive - esp1 & esp2)
  - Pool 1 (22 node production)
mpirun
- to run program "foo" on 3 nodes do:
  mpirun -np 3 foo
- to run "boo -chk -v myrun" on 7 nodes do:
  mpirun -np 7 boo -chk -v myrun
Direct use of POE
- Environment variables (use setenv in (t)csh, export in (k/z)sh)
  - MP_PROCS number of processors
  - MP_RMPOOL 1
  - MP_EUILIB ip for Internet Protocol (IP) or us for User Space (US) Protocol
  - MP_EUIDEVICE en0 for Ethernet or css0 for the HPS. US requires the use of css0. To run (slowly) on a fully utilized machine specify en0
- Examples
  - to run "boo -chk -v myrun" on 3 nodes, using IP communications over Ethernet, do:
    setenv MP_PROCS 3
    setenv MP_RMPOOL 1
    setenv MP_EUILIB ip
    setenv MP_EUIDEVICE en0
    poe ./boo -chk -v myrun
Checking the job status (jm_status)
- jm_status -j (gives a list of all nodes and who is running on them)
- A jm_status -j output line of the form:
  Usage: cpu=DEDICATED adapter=DEDICATED
  means that no other MPI job can run on that node unless it uses IP communications

I/O and storage

temporary disks (/tmp)
Serial and pseudo-parallel I/O. Proper Parallel I/O is not supported yet. (IBM has recently come out with PIOFS.)

General Guidelines on Parallelization

Domain Decomposition
Identification of parallelizable sections
The Prism paradigm

More Information

IBM's InfoExplorer hypertext on-line manuals
- info -l pe will give most relevant information
- info -l essl covers ESSL
- info -l pessl covers PESSL
This document (http://www.cfm.brown.edu/tutorials/sp2.tutorial.html)
Parallel links on the Web (http://www.cfm.brown.edu/tutorials/parallel.links.html)
All CFM users have access to the SP2. If you are not currently a CFM user and would like to use the SP2, contact Henry Robinson at hlr@cfm.brown.edu to arrange for an account and get all of the S-key login details.
Contact e-mail addresses:
- System administration
  - Richard Boes at rcb@cfm.brown.edu
  - Henry Robinson at hlr@cfm.brown.edu
- CRUNCH group user helping with the SP2:
  - Constantinos Evangelinos at ce107@cfm.brown.edu