NOTE: These intructions were updated in June 2006. They may not be kept up-to-date. Refer to the
ScaLAPACK website for more recent and more complete instructions.
ScaLAPACK must be available on your system before Amesos_Scalapack or Amesos_Mumpscan be used. ScaLAPACK is provided in many
high performance libraries, including the Cray Scientific Library (-lsci),
Silicon Graphic's (-lsdsm), Hewlett Packard's MLIB, Intel's MKL. Some IBM scientific libraries also
support ScaLAPACK. Many high performance computer installations include ScaLAPACK. Hence,
it is worth checking to see if ScaLAPACK is already installed on your computer.
Even if ScaLAPACK is available on your system, you may need to build it from the
original sources if you are using a different fortran compiler.
To install ScaLAPACK on your system you will need the BLAS
and the BLACS. The BLAS are almost certainly installed on your
system, but the BLACS are often not installed unless SacaLAPACK is.
To install the BLACS on your system:
- Obtain the distribution from the BLACS website You will need both
blacstester.tgz is also wise.
- Untar these three into the same directory.
- Edit Bmake.inc, starting from one of the templates in BMAKES/Bmake.inc.
- Set BTOPdir to point to the directory that the BLACS are in (generally ends in BLACS)
- Set F77, CC, MPIINCdir and MPILIB.
- MPIINCdir should be set to a directory including mpif.h.
- MPILIB can be left blank if F77=mpif77 and CC=mpicc
- Set PLAT to a character string representing your platform (optional).
- Follow the directions in BLACS/INSTALL/README. The BLACS/INSTALL directory
contains several small routines which will help you set Bmake.inc variables
INTFACE and TRANSCOMM and may warn you of more serious problems with your mpi
implementation. Note: the executables are placed in the EXE subdirectory.
- Type "make mpi" in the BLACS directory to build the BLACS.
- Type "cd TESTING; make" to build the BLACS tester (note, the blacs tester takes close to a minute to compile)
- Run the blacs tester on 4 processes. On many machines, this can be done with:
"cd EXE; mpirun -np 4 xFbtest_MPI-...-0" or "cd EXE; mpirun -np 4 xCbtest_MPI-...-0"
- Don't give up if the BLACS tester fails. It is important that the testers link,
but sometimes blacs installations that cause the BLACS tester to fail still work.
On one system, the BLACS tests all gave this error message:
"MPI Error, rank:0, function:MPI_TYPE_INDEXED, Invalid count argument"
but the ScaLAPACK and Amesos_ScaLAPACK installation tests both passed.
To install ScaLAPACK on your system, install the BLACS first(see above), then:
- Obtain the latest distribution from the SCALAPACK website. As of May 2006, the latest distribution is scalapack-1.7.3.tgz, but you should check for patches and later distributions.
- Untar the ScaLAPACK code: "tar xzf scalapack-1.7.3.tgz". Typically this is done in the same directory that you untarred the BLACS code (it will create a scalapack-1.7.3 directory).
- Edit SLmake.inc.
- home to the directory that the ScaLAPACK code resides in. For version 1.7.3, this will end in scalapack-1.7.3. For earlier versions, it tyically ends in SCALAPACK.
- Set PLAT to the same staing that you set it to for the BLACS in the Bmake.inc file (optional).
- Set BLACSdir to the directory that the BLACS library files are in (typically ends in BLACS/LIB)
- Set F77, CC. Typically (almost always) these are the same compilers used to compile the BLACS.
- Set CDEFS to the flags you wish to pass to the C compiler (typically this includes the flags you set in
INTFACE in Bmake.inc).
- Set SMPLIB to point to your mpi library (typically -lmpi). May be left blank if you use F77=mpif77 and CC=mpicc
- Set BLASLIB to point to the BLAS library.
- Set BLACSFINIT, BLACSCINIT and BLACSLIB. If you set BLACSdir and PLAT right, you may not have to change these.
- Type: "make" in the same directory that SLmake.inc is in to build the ScaLAPACK library. Takes several minutes.
- Check to see that libscalapacl.a was built in this directory.
- Type: "cd TESTING/LIN; make" to build the linear solver testers. Takes several minutes.
- Type: "cd .." to change back to the TESTING directory. Then run xdlu < LU.dat on four processes.
On many computers this can be done with "mpirun -np 4 xdlu The results should end with:
Finished 240 tests, with the following results:
240 tests completed and passed residual checks.
0 tests completed and failed residual checks.
0 tests skipped because of illegal input values.
END OF TESTS.
To build Amesos-Scalapack:
- Update your configure invocation script by, for example, adding something along the following lines:
- --enable-amesos-scalapack F77=gfortran
--with-libs="/home/username/Trilinos3PL/SCALAPACK/scalapack-1.7.3/libscalapack.a /home/username/Trilinos3PL/SCALAPACK/BLACS/LIB/blacsF77init_MPI-SGI5-0.a /home/username/Trilinos3PL/SCALAPACK/BLACS/LIB/blacs_MPI-SGI5-0.a "
- The lapack and blas libraries may or may not need to be listed explicitly and, in some cases,
may need to be listed in the --with-libs line. If you are having trouble, you may want to try
adding CXXFLAGS=-g to speed compilation.
- Common problems:
Both undefined externals and multiply defined externals can prevent Trilions from either configuring or
building. Two routines, xerbla and lsame, are included in multiple libraries. In general, this should not
be a problem, but in some cases, it can be.
xerbla and pxerbla the only routines in the scalapack/lapack/blas suite that print. Different fortran
compilers use different routines for printing. g77 uses: s_wsfe, do_fio, e_wsfe and s_stop. ifort uses:
for_write_seq_fmt, for_write_seq_fmt_xmit and for_stop_core. g95 uses: g95_st_write, g95_stop_blank and others.
gfortran uses: gfortran_st_write, gfortran_stop_numeric and others. If any of these routines are listed as undefined by the link step, you may be mixing codes compiled by more than one fortran compiler.
Compiling xerbla.f with
the fortran compiler that you intend to use and examining it with "nm xerbla.o" will
tell you which routines your fortran compiler uses in its runtime library.
If the blas library that you are using were compiled by a compiler other
than the compiler that you are using, you will either have to link
fortran run time libraries from both compilers (not recommended) or
make sure that a xerbla.o built by the fortran compiler that you are
using gets linked in prior to that blas library. This can be accomplished by listing a library with the
correctly compiled xerbla.o in the --with-blas= switch. For example, in one implementation, I added
libscalapack.a to the front of the --with-blas command because it had a xerbla.o complied with the fortran
compiler that I wanted to use (an f90 compiler for MUMPS). Hence, my configuration invocation included:
-L/home/kstanley/Linux_IA64Itan_2/lib -lcblas -lf77blas -latlas " In some cases, it may be necessary (or expedient) to remove xerbla.o and lsame.o from a pre-compiled library. This can often be done with: "ar d libname.a xerbla.o; ar d libname.a lsame.o"
Problems with incompatible blas libraries often show up as configuration or link errors in the teuchos package as that is the first package to be configured and built in Trilinns.
Undefined externals in the link phase mean either that object files
which you don't want are being linked in (typically xerbla.o produced
by a different fortran compiler than the one you are using) or that
you are not linking needed files. If the link step is including files that
you do not want, get rid of them, either by eliminating them from the link line, or
if that is not possible (for example, a xerbla.o file in a library that you wish to use but
which was compiled by a different fortran compiler than the one you want to use) you will need to
ensure that the offending file is not linked in by ensuring that the entry point which it satisfies is
linked before it in the link step. See the description above for how we convinced the linker to
include xerbla.o from libscalack.a instead of the blas.
If the undefined externals stem from a file that you do wish to
include in the link, you will need to include the libraries which
satisfy these externals. Linking fortran codes in a link performed by
a call to a C++ compiler (as Trilinos typically does) requires that
the fortran libraries be listed explicitly. If you don't know which files
contain the fortran runtime libraries that you need, link a trivial fortran code using te fortran compiler
as the linker, then add "-v" to that
link step to show the files included in the link. Alternatively, you can use
which will usually cause the linker to print a link map.
If the names of routines that are typically written in fortran,
such as scalapack, lapack or blas routines, are not found by the
linker, it is possible that the wrong form of external names is being
used. Some fortran compilers use all capital letters for external
names, most use lowercase name with a single underscore appended to
the end. But, some use lowercase without appending an underscore and
some fortran compilers append an additional underscore to routines
which include an underscore in their names (like sl_init or
blacs_gridinit). "nm" can be used to
determine which convention is used in the lapack, scalapack and blas
libraries that you are using (if
the three libraries do not all stick to the same naming conventino for the entry points,
you will probably need to recompile one or more of those
libraries). If the Trilinos link step is failing as a result of
different fortran naming conventions, Trilinos may be using a different
compiler than the one that was used to compile the libraries that you are using.
If the link works, but the execution fails because a shared library file could not be found,
you will need to change the environment variable LD_LIBRARY_PATH. This is typically done in
.cshrc, .profile or .bashrc.
If the configuration fails, the file packages/failing_package/config.log will contain some
information about the failure. Often, repeating the last attempt to compile conftest.c as shown
in config.log (both conftest.cc and the command used to compile and link it are saved in config.log)
will provide useful information. I typically play with this until that compilation and link works before
trying another Trilinos configuration.