Some option of PETSc

This is reminded me not for all user.
1cup:
./run.exe -inputfile input.txt -ksp_type preonly -pc_type lu -pc_factor_shift_nonzero -snes_rtol 1.e-5 -snes_max_it 6 -iterations 5000000 -snes_monitor -dump_data_cycle 200 -CONSTANT_TIME_STEP -dump_data 3020

PETSc with intel MKL

Fastest option:
./config/configure.py --with-clanguage=c++ --with-fc=0 --with-cc=mpicc --with-cxx=mpicxx --with-debugging=0 --with-vendor-compilers=intel --with-blas-lapack-dir=/opt/intel/Compiler/11.0/083/mkl/lib/32

2.3.3
./config/configure.py --with-clanguage=c++ --with-fc=0 --with-cc=mpicc --with-cxx=mpicxx --with-debugging=0 --with-vendor-compilers=intel --with-gnu-compilers=0 --with-x=0 --with-mpi-dir=/opt/mpich-intel/ --CXXFLAGS="-Kc++ -O2" --CFLAGS=-O2 --CPPFLAGS=-O2 --CXX_CXXFLAGS="-Kc++ -O2" --with-blas-lapack-dir=/opt/intel/mkl/10.1.1.019/


3.0.0
./config/configure.py --with-clanguage=cxx --with-fortran=0 --with-windows-graphics=0 --with-cc=/opt/mpich-intel/bin/mpicc --with-cxx=/opt/mpich-intel/bin/mpicxx --with-gnu-compilers=0 --with-vendor-compilers=intel --CC=/opt/mpich-intel/bin/mpicc --CXX=/opt/mpich-intel/bin/mpicxx --with-fc=0 --with-debugging=no --CFLAGS='-O2' --CXXFLAGS='-O2' --CXX_CXXFLAGS='-O2' --with-mpi-dir=/opt/mpich-intel --with-x=0 --with-x11=0 --with-blas-lapack-dir=/opt/intel/mkl/10.1.1.019/lib/64

running fast:
./config/configure.py --with-clanguage=cxx --with-fortran=0 --with-windows-graphics=0 --with-gnu-compilers=0 --with-vendor-compilers=intel --with-debugging=0 --with-errorchecking=0 --with-c++-support --with-log=0 --with-info=0 --CFLAGS=-O3 --CXXFLAGS=-O3 --with-mpi-dir=/home/benvar/lib/mpich2 --with-x=0 --with-x11=0 --download-c-blas-lapack=1

How to draw and view data when PETSc running

Example solve AX=B
vec X;

• In source code

VecView(Vec vec,PetscViewer viewer)
VecView(X,PETSC_VIEW_STDOUT_SELF)

• In command

VecViewFromOptions(Vec vec, char *title)
VecViewFromOptions(X,"TEST")


petscmpiexec -n 2 ./a.out -vec_view_draw -draw_pause 1

Distribution Data (DA)

DAGetInfo
Gets information about a given distributed array.
Synopsis
#include "petscda.h"
PetscErrorCode PETSCDM_DLLEXPORT DAGetInfo(DA da,PetscInt *dim,PetscInt *M,PetscInt *N,PetscInt *P,PetscInt *m,PetscInt *n,PetscInt *p,PetscInt *dof,PetscInt *s,DAPeriodicType *wrap,DAStencilType *st)
Input Parameter
da -the distributed array
Output Parameters
dim - dimension of the distributed array (1, 2, or 3)
M, N, P - global dimension in each direction of the array
m, n, p - corresponding number of procs in each dimension
dof - number of degrees of freedom per node
s - stencil width
wrap - type of periodicity, one of DA_NONPERIODIC, DA_XPERIODIC, DA_YPERIODIC, DA_XYPERIODIC, DA_XYZPERIODIC, DA_XZPERIODIC, DA_YZPERIODIC,DA_ZPERIODIC
st - stencil type, either DA_STENCIL_STAR or DA_STENCIL_BOX
Note
Use PETSC_NULL (PETSC_NULL_INTEGER in Fortran) in place of any output parameter that is not of interest.

DAGetCorners
Returns the global (x,y,z) indices of the lower left corner of the local region, excluding ghost points.
Synopsis
#include "petscda.h"
PetscErrorCode PETSCDM_DLLEXPORT DAGetCorners(DA da,PetscInt *x,PetscInt *y,PetscInt *z,PetscInt *m,PetscInt *n,PetscInt *p)
Input Parameter
da -the distributed array
Output Parameters
x,y,z - the corner indices (where y and z are optional; these are used for 2D and 3D problems)
m,n,p - widths in the corresponding directions (where n and p are optional; these are used for 2D and 3D problems)
Note
The corner information is independent of the number of degrees of freedom per node set with the DACreateXX() routine. Thus the x, y, z, and m, n, p can be thought of as coordinates on a logical grid, where each grid point has (potentially) several degrees of freedom. Any of y, z, n, and p can be passed in as PETSC_NULL if not needed.

PetscBarrier

PetscBarrier
Blocks until this routine is executed by all processors owning the object A.
Synopsis
#include "petsc.h"
PetscErrorCode PetscBarrier(PetscObject obj)
Input Parameters
A -PETSc object (Mat, Vec, IS, SNES etc...) Must be caste with a (PetscObject), can use PETSC_NULL (for MPI_COMM_WORLD)

Easy usage: PetscBarrier(PETSC_NULL)

vector function in PETSc

VecCreate
Creates an empty vector object
Synopsis
#include "petscvec.h"
VecCreate(MPI_Comm comm, Vec *vec)

VecSetSizes
Sets the local and global sizes, and checks to determine compatibility
Synopsis
#include "petscvec.h"
PetscErrorCode PETSCVEC_DLLEXPORT VecSetSizes(Vec v, PetscInt n, PetscInt N)
Parameter
v-the vector
n-localsize
N-Globalsize

MPI function in PETCs

MPI_Comm_rank

#include "mpi.h"
int MPI_Comm_rank ( MPI_Comm comm, int *rank)
Determines the rank of the calling process in the communicator

Sone definition of PETSc parameter

• PetscMPIInt = int {typedef int PetscMPIInt}
• PetscErrorCode = int {typedef int PetscErrorCode}
• PetscInt = int {no 64bit }
• PetscInt = long long {64bit}
• PetscErrorCode = int {typedef int PetscErrorCode}
  

How to compiler PETSc's program and run

Makefile:

include ${PETSC_DIR}/bmake/common/base
include ${PETSC_DIR}/bmake/common/test
print.out: p-hellow.o
${CLINKER} -o print.out p-hellow.o ${PETSC_LIB}

Note: It will find correspond file name automatically.
For example: p-hellow.o -> p-hellow.cpp

runfile:
${PETSC_DIR}/bin/petscmpiexec -np 2 ./print.out
OR
mpirun -np 2 ./print.out

How to install or setup PETSc

Environment: mpich-1.2.7p1 (/usr/opt/mpich)
Intel MKL on Windows or Linux (/usr/opt/int/mkl)
Assume I had install MPICH and BLAS & LAPACK.
Therefore, if you want to install single computer PETSc without MPICH

1. PETSC_DIR=$PWD; export PETSC_DIR
2. ./config/configure.py --with-cc=/opt/intel/cc/10.0.14/bin/icc --with-cxx=/opt/intel/cc/10.0.14/bin/icpc --with-fc=/opt/intel/fc/10.0.14/bin/ifort --with-blas-lapack-dir=/usr/opt/intel/mkl/10.0.1.014 --with-mpi=0
3. make all
4. make test
   

• Install PETSc with MPICH

1. PETSC_DIR=$PWD; export PETSC_DIR
2. ./config/configure.py --with-blas-lapack-dir=/usr/opt/intel/mkl/10.0.1.014 --with-mpi-dir=/usr/opt/mpich --with-clanguage=cxx --with-cxx=/usr/opt/mpich/bin/mpicxx
3. make
   
4. make test

BLAS/LAPACK: these packages provide some basic numeric kernels used by PETSc.

• /usr/lib/libblas.a,liblapack.a
• Intel MKL on Windows or Linux
• sunperf on solaris
• VecLib on Macs
• IBM ESSL

Alternatively one can use other externalpackages installation options like the following.

• --with-blas-lapack-lib=libsunperf.a
  

• --with-blas-lib=libblas.a --with-lapack-lib=liblapack.a
• --with-blas-lapack-dir=opt/intel/mkl72

If using MPICH which is already installed [perhaps using myrinet/gm] then use:
. /config/configure.py --with-mpi-dir=/path-to-mpich-install