Composyx Composyx weekly benchmark on plafrim: experimental setup
Table of Contents
Back to index.
See the Experimental setup
1. Test case description
We consider a stationary heterogeneous diffusion equation (or Darcy equation) in a 3D stratified medium \(\nabla \cdot (k\nabla u) = 1\). The equation is discretized using the Finite Element Method. The matrices and right hand side are generated with genfem.
This benchmark is a weak scaling test with 18, 36, 72 and 144 subdomains. Each subdomain consists of a \(30 \times 30 \times 30\) cube. In the Y-direction, conductivity alternates every 5 elements, with an Heterogeneity of 1000.
Figure 1: One subdomain, a cube of edge 30, conductivity \(k=1\) in blue and \(k=1000\) in red
For the baton test case, the subdomains are put one against the other in the X-dimension.
Figure 2: Baton test case, with \(6 \times 1 \times 1\) subdomains
For the cuboid test case, the subdomains are put so as to form the "most cubic" shape possible.
Figure 3: Cuboid test case, with \(4 \times 3 \times 2\) subdomains
2. Baton test case, on the input matrix K
2.1. openblas
2.2. mkl
3. Baton test case, on the Schur complement matrix S
3.1. openblas
3.2. mkl
4. Experiments in command line form
4.1. Preliminaries
rm -rf build && mkdir build && cd build
4.2. Command (Example)
For 36 subdomains with AS/S preconditionners using openblas
# creating Makefile guix shell --pure -D composyx slurm eigen armadillo scalapack zlib -- cmake .. -DCMAKE_BUILD_TYPE=Release -DCOMPOSYX_COMPILE_EXAMPLES=ON -DCOMPOSYX_COMPILE_TESTS=OFF -DCOMPOSYX_DEV_WARNINGS=OFF # compiling binary target guix shell --pure -D composyx slurm eigen armadillo scalapack zlib -- make composyx_driver_cg cd src/examples/ cat <<EOF > input.in n_subdomains: 36 max_iter: 500 # Working on the Schur complement use_schur: 1 # Path to partitions partitions: /home/gitlab-compose/baton_30_newformat # Partiton type partition_type: baton # Type of preconditioner precond: AS # Pastix or Mumps direct_solver: Pastix tolerance: 1e-6 EOF cat <<EOF > batch.batch #!/bin/bash #SBATCH --job-name=composyx #SBATCH --nodes=1 #SBATCH --ntasks=36 #SBATCH --ntasks-per-node=36 #SBATCH --cpus-per-task=1 #SBATCH --time=00:10:00 #SBATCH --output=slurm.out #SBATCH --error=slurm.out #SBATCH --exclusive #SBATCH --constraint bora module load mpi/openmpi/4.1.1 export OMPI_MCA_pml='^ucx' guix shell --pure --preserve='^SLURM|^OMPI' -D composyx slurm eigen armadillo scalapack zlib -- mpiexec -n \${SLURM_NPROCS} ./composyx_driver_cg input.in EOF sbatch batch.batch
For 36 subdomains with AS/S preconditionners using mkl, you should add to the guix environment:
--with-input=mumps-openmpi=mumps-mkl-openmpi --with-input=openblas=mkl
Output is in slurm.out, errors are in slurm.err.
5. Source
Run date:
sam. 21 sept. 2024 01:14:40 CEST
Commit:
f9e360e6651df44fccc2ad6fc2d1f8ff7d225330
Channels:
(list (channel (name 'guix-hpc-non-free) (url "https://gitlab.inria.fr/guix-hpc/guix-hpc-non-free.git") (branch "master") (commit "5c78c7262f902c807e66852d326076d183b36b9c")) (channel (name 'guix) (url "https://git.savannah.gnu.org/git/guix.git") (branch "master") (commit "2c54c2db410ebdda8cd71716315e4ea4d31befbd") (introduction (make-channel-introduction "9edb3f66fd807b096b48283debdcddccfea34bad" (openpgp-fingerprint "BBB0 2DDF 2CEA F6A8 0D1D E643 A2A0 6DF2 A33A 54FA")))) (channel (name 'guix-hpc) (url "https://gitlab.inria.fr/guix-hpc/guix-hpc.git") (branch "master") (commit "ac0563ef7ec5c49a4a79e533dcb0d7409eb26fd4")) (channel (name 'guix-science-nonfree) (url "https://github.com/guix-science/guix-science-nonfree.git") (branch "master") (commit "3413215081da370823c98e79445d8cbfd37531db") (introduction (make-channel-introduction "58661b110325fd5d9b40e6f0177cc486a615817e" (openpgp-fingerprint "CA4F 8CF4 37D7 478F DA05 5FD4 4213 7701 1A37 8446")))) (channel (name 'guix-science) (url "https://github.com/guix-science/guix-science.git") (branch "master") (commit "3736b80cf569511cc0a13f87cb03411761290dd8") (introduction (make-channel-introduction "b1fe5aaff3ab48e798a4cce02f0212bc91f423dc" (openpgp-fingerprint "CA4F 8CF4 37D7 478F DA05 5FD4 4213 7701 1A37 8446")))) (channel (name 'guix-past) (url "https://gitlab.inria.fr/guix-hpc/guix-past") (branch "master") (commit "792d830606345bddf5712252a1ed4fd87ee55c42") (introduction (make-channel-introduction "0c119db2ea86a389769f4d2b9c6f5c41c027e336" (openpgp-fingerprint "3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5")))))
results.csv
NbDomains,preconditioner,use_schur,blas,DirectSolver,nIter,CoarseEigenSolve,DirectSolve,CoarsePcdSetup,IterativeSolve,LocalPcdSetup,SchurComputation,TotalTime,RunTime,GlobMatOrder,robin_weight,threadspertask 36,diag,0,openblas,Mumps,2907,0,0,0,6.10487,0.00109815,0,6.11731,6.6241,1038876,1.0,1 36,diag,0,mkl,Mumps,2907,0,0,0,6.20122,0.0010703,0,6.21354,6.77482,1038876,1.0,1 36,diag,1,openblas,Mumps,445,0,0.0511532,0,2.07254,0.000211748,1.37708,3.50242,4.02217,1038876,1.0,1 36,diag,1,mkl,Mumps,445,0,0.0408961,0,2.08148,0.000166859,1.2757,3.4026,4.04323,1038876,1.0,1 36,AS,0,openblas,Mumps,353,0,0,0,12.5081,0.0827824,0,12.5201,13.0658,1038876,1.0,1 36,AS,0,mkl,Mumps,353,0,0,0,11.8411,0.0838694,0,11.8555,12.4539,1038876,1.0,1 36,AS,1,openblas,Mumps,59,0,0.0507074,0,1.01046,0.0829487,1.39068,2.45522,2.96013,1038876,1.0,1 36,AS,1,mkl,Mumps,59,0,0.0400108,0,0.744918,0.0846294,1.2888,2.07827,3.46026,1038876,1.0,1