1. Itanium2 Nodes

• a54-1727-641 (head node, 1 1GHz Itanium2, 1GB RAM)

• a54-1727-642, a54-1727-643 (2 1GHz Itanium2, 16GB RAM)

The following nodes are running the Red Hat EL AS 3 (Update 3) provided kernel:

• aE34-500-641 (head node, 1 1GHz Itanium2, 1GB RAM)

• aE34-500-642 (2 1GHz Itanium2, 16GB RAM)

• aE34-500-643 (2 1GHz Itanium2, 8GB RAM)

• a48-206-641 (head node, 1 1GHz Itanium2, 1GB RAM)

• a48-206-642, a48-206-643 (2 1GHz Itanium2, 16GB RAM)

There exist another 3 nodes that are not visible currently.

2. IA64 Software

2.1. Interpreters

• Perl 5.8.0 is installed from RPM

• Python 2.2.3 is installed from RPM

• TCL/TK 8.4.5 is installed from RPM

• Java SDK 1.3.2_06 installed in /usr/local/pkg-ia64/jdk/j2sdk1.4.2_06

• • module add jdk

• IBM Java SDK 1.4.2 is installed from RPM

2.1.1. Extra Tools

• SWIG 1.3.19 is installed from RPM

• tDOM 0.8.1 installed in /usr/local/pkg-ia64/tdom/tdom-0.8.1

• • module add tdom

2.2. Compilers

• The latest Intel 8.1 compilers are installed in /usr/local/pkg-ia64/{ecc|efc} for the C/C++, Fortran compilers respectively.

• • module add ecc etc.

  • Call as ecc/icc, ecpc/icpc, efc/ifort respectively. The names starting with "e" are being phased out.

• The latest Open Research Compiler for ItaniumTM Processor Family (IPF-ORC is installed in /usr/local/pkg-ia64/orc.

• • module add orc

  • Call as orcc, orc++/orCC and orf90

  • The flags can can seen by calling the compiler with the --help flag. They are like the MIPSPro compiler flags.

  • • Do not use the Ofast or any flag that uses IPO as it is currently broken.

  • It requires pieces of GCC 2.96 backend.

  • Claims very high performance

• The latest Open64 Compiler source code has been downloaded but needs either GCC 2.95 or GCC 3.3/3.4 (with patching of the Open64 source code) to build. This will have to wait.

• GCC 2.96 is installed from RPM

• • Call as gcc296, g++296.

  • No Fortran 77.

• GCC 3.2.3 is installed from RPM

• • Call as gcc, g++, g77, gnat, gcj etc.

• GCC 3.5-SSA (known as GCC4 nowadays) quite early version (03 Aug 2004) is installed from RPM

• • Call as gcc-ssa, g++-ssa, g77-ssa, gcj-ssa etc.

  • Appears to offer the best performance of the GCC versions

2.3. Libraries

2.3.1. BLAS and LAPACK

• Unoptimized Netlib BLAS and LAPACK 3.0 are installed from RPM.

• • Link with -lblas and -llapack -lblas for BLAS only and LAPACK/BLAS respectively.

• GOTO BLAS 0.95 in /usr/local/pkg-ia64/goto/goto-0.95/

• • Add the appropriate library path and link with -lgoto or -lgotop for single threaded/pthreaded code.

  • Highest DGEMM performance

  • May need to also compile and link in xerbla.c or xerbla.f in /usr/local/pkg-ia64/goto

• ATLAS 3.6.0 BLAS and bit of LAPACK in /usr/local/pkg-ia64/ATLAS/3.6.0/Linux_IA64Itan2

• • Add the appropriate library path and link with -lcblas -lf77blas -latlas for single threaded code

  • To get some of LAPACK add -llapack in front, be careful not to confuse with system lapack library. To use both the ATLAS and the system library for the rest of the LAPACK calls, link in

  the ATLAS library with the full pathname /usr/local/pkg-ia64/ATLAS/3.6.0/Linux_IA64Itan2/lib/liblapack.a ahead of the system library.

• Intel MKL 7.0 in /usr/local/pkg-ia64/mkl (also for IA32 in /usr/local/pkg/mkl)

• • We need a license for ACES, maybe also upgrade to v. 7.2? (a PI whose work uses it maybe?)

  • Add the appropriate library path and link with -lmkl (dynamic), -lmkl_ipf (static)

  • For LAPACK link with -lmkl_lapack32 (dynamic), -lmkl_lapack (static)

2.3.2. Generic Math Libraries

• Intel IPP 4.1 in /usr/local/pkg-ia64/ipp (also for IA32 in /usr/local/pkg/ipp)

• GNU Scientific Library (GSL) 1.4.2 is installed from RPM

2.3.3. FFT Libraries

• FFTW

• • 3.0.1 in /usr/local/pkg-ia64/fftw/fftw-3.0.1/intel

  • 2.1.5 in /usr/local/pkg-ia64/fftw/fftw-2.1.5/intel

  • • MPI versions of the 2.1.5 library need to be installed

• Intel MKL also includes FFT routines

• • We need a license for ACES (a PI whose work uses it maybe?)

• Intel IPP also includes FFT routines (probably the fastest but less versatile)

• • We need a license for ACES (a PI whose work uses it maybe?)

3. Queueing

• Torque needs to be added.

  • At this moment access to the Itanium2 nodes is not controlled by a queueing system and users may inadvertently run programs competing for the same CPU and memory resources on a given node. Please be considerate and careful when using these nodes.
  • Even after the introduction of IA64 queues we expect to keep the single cpu nodes as development/interactive nodes as they have the development environment for IA64 code generation.

4. Parallel Frameworks

• SCore 5.8.2 is installed in /usr/local/pkg-ia64/score/score-5.8.2 and contains:

• • PMv2

  • PMv2 is a low level communication library for cluster computing. The PMv2 API (Application Program Interface) is carefully designed so that many types of networks and shared memory are accessed in a uniform way for cluster computing. PMv2 drivers for Ethernet, UDP, Shmem have been installed.

  • SCore-D

  • A user-level global operating system to utilize cluster resources, such as processors, networks. Various PM network devices are managed and utilized by SCore-D. SMP clusters, heterogeneous clusters as well as homogeneous clusters are supported. SCore-D provides a flexible parallel job scheduling scheme. User parallel jobs are time-shared and space-shared simultaneously and effectively. Preemptive consistent checkpointing is provided by SCore-D, and any user programs can be checkpointed by only specifying a checkpoint interval time.

  • MPICH-SCore (1.2.0, 1.2.4 and 1.2.5) on top of PMv2

  • SCASH

  • A software DSM (Distributed Shared Memory) system using PMv2. It employs the Lazy Release Consistency model with both update and invalidate protocols.

  • MPC++

  • A multi-threaded C++ using the C++ template feature. It provides synchronous/asynchronous remote function invocation facilities, synchronization structures, global pointer, and other such functions.

  • TACO

  • TACO (Topologies and Collections) is a general purpose distributed object platform based on C++ templates. TACO provides a powerful dynamic group concept as the basic means for data-parallel programming. Entire distributed objects groups can be created, destroyed, selectively cloned and manipulated by various parallel collective operations. Thus high-level parallel programming becomes possible without any language extensions. Since TACO is implemented as a thin library layer on top of the MTTL it yields an excellent performance on cluster architectures.

    • SCOUT

    • The scout program provides a SIMD-style remote UNIX shell environment. Any UNIX command can be executed on cluster hosts in parallel.
  • For SCore to work, apart from the SCore-aware kernel, some further setup is required that also apparently needs fully qualified, even if ficticious, domain names for the nodes. This is not the case currently on the IA64 cluster nodes.

  • module add score

5. Message Passing

• MPI builds needed (modules are still needed)

• • LAM 7.1.1 (needs Torque to be installed first)

  • MPICH-VMI 1.2.5 and VMI2.0 installed in /usr/local/pkg-ia64/mpich-vmi/mpich-vmi-1.2.5/intel

  • MPICH 1.2.6 installed in /usr/local/pkg-ia64/mpich/mpich-1.2.6/intel

  • MPICH2 1.0 installed in /usr/local/pkg-ia64/mpich2/mpich2-1.0/intel

• SCore MPICH:

• • /usr/local/pkg-ia64/score/score-5.8.2/mpi/mpich-1.2.{0|4|5}/ia64-redhat-linux2_4_{gnu|intel}

• MPI tools to be added:

• • mpiP 2.7

6. Math Tools

• GNU Octave 2.1.50 is installed from RPM

• • Matlab-like, with a compatibility set of options. It may be as simple as running octave --traditional and other changes mentioned in the link above to use your Matlab scripts. Otherwise you may need to use:

  • Matlab Compatibility Database are installed in /usr/local/pkg-ia64/matcomp/matcomp-2001.02.25. As these scripts are 3+ years old, their functionality may already be in the octave version installed so do not do the following unless you discover that it provides you Matlab compatibility octave as installed does not have:

  • • Modify .octaverc in your home directory to include:

LOADPATH = [ '/usr/local/pkg/matcomp/matcomp-2001.02.25/site/oct:/usr/local/pkg/matcomp/matcomp-2001.02.25/site//:', LOADPATH ];
EXEC_PATH = [ '/usr/local/pkg/matcomp/matcomp-2001.02.25/site/bin:', EXEC_PATH ];

• Octave-Forge scripts (like Matlab toolkits) are installed in /usr/local/pkg-ia64/octave-forge/octave-forge-2004.11.16

• • Modify .octaverc in your home directory to include:

LOADPATH = [ '/usr/local/pkg-ia64/octave-forge/octave-forge-2004.11.16/site/oct:/usr/local/pkg-ia64/octave-forge/octave-forge-2004.11.16/site//:',LOADPATH ];
EXEC_PATH = [ '/usr/local/pkg-ia64/octave-forge/octrave-forge-2004.11.16/site/bin:',EXEC_PATH ];

7. Debuggers

• GNU Debugger 6.0post is installed from RPM

• Intel Debugger 8.1-042 is installed in /usr/local/pkg-ia64/idb

• • module add idb

• Data Display Debugger (DDD) is installed from RPM. This graphical debugger works with both gdb and idb as the back end debugger.

8. Debugging Tools

• Electric Fence is installed from RPM

• Strace is installed from RPM

9. Performance Tools

• perfmon installed in custom kernel

• • libpfm 2.0 installed in /usr/local/pkg-ia64/libpfm/libpfm-2.0

  • • module add libpfm

  • pfmon 2.0 installed in /usr/local/pkg-ia64/pfmon/pfmon-2.0

  • • module add pfmon

• PAPI

• • PAPI3 v. 3.0.7 is installed in /usr/local/pkg-ia64/papi/papi-3.0.7

  • • module add papi/3.0.7

    • [ papiex] is installed in /usr/local/pkg-ia64/papiex/papi-0.9

    • • module add papiex

  • PAPI2 v. 2.3.4.3 is installed in /usr/local/pkg-ia64/papi/papi-2.4.3.4

  • • module add papi/2.3.4.3

    • PerfBench 5.2 is installed in /usr/local/pkg-ia64/perfbench/perfbench-5.2

    • • module add perfbench

• PerfSuite 0.6.1rc1 is installed:

• • build with PAPI3 in /usr/local/pkg-ia64/perfsuite/perfsuite-0.6.1rc1/papi3

  • • module add perfsuite/papi3

  • build with PAPI2 in /usr/local/pkg-ia64/perfsuite/perfsuite-0.6.1rc1/papi2

  • • module add perfsuite/papi2

• Qprof is installed in /usr/local/pkg-ia64/qprof/qprof-0.5

• • module add qprof

• HPCToolkit is installed in /usr/local/pkg-ia64/HPCToolkit/HPCToolkit-4.0.0

• • module add hpctoolkit

• Pablo is under consideration to be installed

• VProf is under consideration to be installed

• TAU is under consideration to be installed