\( \newcommand{\blu}[1]{{\color{blue}#1}} \newcommand{\red}[1]{{\color{red}#1}} \newcommand{\grn}[1]{{\color{green!50!black}#1}} \newcommand{\local}{_i} \newcommand{\inv}{^{-1}} % Index for interface and interior \newcommand{\G}{\Gamma} \newcommand{\Gi}{\Gamma_i} \newcommand{\I}{{\cal I}} \newcommand{\Ii}{\I_i} % Matrix A \newcommand{\A}{{\cal A}} \newcommand{\Ai}{\A\local} \newcommand{\Aj}{\A_j} \newcommand{\Aib}{\bar{\A}\local} \newcommand{\AII}{\A_{\I\I}} \newcommand{\AIG}{\A_{\I\G}} \newcommand{\AGI}{\A_{\G\I}} \newcommand{\AGG}{\A_{\G\G}} \newcommand{\AIiIi}{\A_{\Ii\Ii}} \newcommand{\AIiGi}{\A_{\Ii\Gi}} \newcommand{\AGiIi}{\A_{\Gi\Ii}} \newcommand{\AGiGi}{\A_{\Gi\Gi}} \newcommand{\AGiGiw} {{\ensuremath{\A_{\Gi\Gi}^w}}} \newcommand{\AIIi}{\AII\local} \newcommand{\AIGi}{\AIG\local} \newcommand{\AGIi}{\AGI\local} \newcommand{\AGGi}{\AGG\local} \newcommand{\Ab}{\bar{\A}} \newcommand{\Ah}{{\widehat{\A}}} \newcommand{\Aih}{{\Ah\local}} \newcommand{\At}{{\widetilde{\A}}} \newcommand{\Ait}{{\At\local}} \newcommand{\Ao}{\A_0} \newcommand{\Aot}{\At_0} \newcommand{\Aob}{\Ab_0} \newcommand{\AiNN}{\Ai^{(NN)}{}} \newcommand{\AitNN}{\Ait^{(NN)}{}} \newcommand{\AiAS}{\Ai^{(AS)}{}} \newcommand{\AitAS}{\Ait^{(AS)}{}} % Matrix S \renewcommand{\S}{{\cal S}} \newcommand{\Si}{\S\local} \newcommand{\Sb}{\bar{\S}} \newcommand{\Sib}{\Sb\local} \newcommand{\Sh}{{\widehat{\S}}} \newcommand{\Sih}{{\Sh\local}} \newcommand{\St}{{\widetilde{\S}}} \newcommand{\Sit}{{\St\local}} \newcommand{\So}{\S_0} \newcommand{\Soi}{\S_0^i} \newcommand{\Sot}{\St_0} \newcommand{\Sob}{\Sb_0} \newcommand{\SiNN}{\Si^{(NN)}{}} \newcommand{\SitNN}{\Sit^{(NN)}{}} \newcommand{\SiAS}{\Si^{(AS)}{}} \newcommand{\SitAS}{\Sit^{(AS)}{}} % Matrix K \newcommand{\K}{{\cal K}} \newcommand{\Ki}{\K\local} \newcommand{\Kb}{\bar{\K}} \newcommand{\Kib}{\Kb\local} \newcommand{\Kh}{{\widehat{\K}}} \newcommand{\Kih}{{\Kh\local}} \newcommand{\Kt}{{\widetilde{\K}}} \newcommand{\Kit}{{\Kt\local}} \newcommand{\Ko}{\K_0} \newcommand{\Kot}{\Kt_0} \newcommand{\Kob}{\Kb_0} \newcommand{\KiNN}{\Ki^{(NN)}{}} \newcommand{\KitNN}{\Kit^{(NN)}{}} \newcommand{\KiAS}{\Ki^{(AS)}{}} \newcommand{\KitAS}{\Kit^{(AS)}{}} \newcommand{\KII}{\K_{\I\I}} \newcommand{\KIG}{\K_{\I\G}} \newcommand{\KGI}{\K_{\G\I}} \newcommand{\KGG}{\K_{\G\G}} \newcommand{\KIiIi}{\K_{\Ii\Ii}} \newcommand{\KIiGi}{\K_{\Ii\Gi}} \newcommand{\KGiIi}{\K_{\Gi\Ii}} \newcommand{\KGiGi}{\K_{\Gi\Gi}} \newcommand{\KIIi}{\KII\local} \newcommand{\KIGi}{\KIG\local} \newcommand{\KGIi}{\KGI\local} \newcommand{\KGGi}{\KGG\local} \newcommand{\KGiGiw} {{\ensuremath{\K_{\Gi\Gi}^w}}} % Matrix B \newcommand{\B}{{\cal B}} \newcommand{\Bi}{\B\local} \newcommand{\Bib}{\widehat{\B}\local} \newcommand{\Bob}{\widehat{\B}_0} % Matrix C \newcommand{\C}{{\cal C}} % Matrix T \newcommand{\T}{{\cal T}} \newcommand{\Ti}{{\T\local}} % Vectors \newcommand{\uI}{u_\I} \newcommand{\uG}{u_\G} \newcommand{\xI}{x_\I} \newcommand{\xG}{x_\G} \newcommand{\bI}{b_\I} \newcommand{\bG}{b_\G} \newcommand{\fI}{f_\I} \newcommand{\fG}{f_\G} \newcommand{\ftG}{\widetilde f_\G} \newcommand{\ftGi}{\widetilde f_\Gi^{(i)}} \newcommand{\ftGj}{\widetilde f_\Gj^{(j)}} \)

Composyx Composyx weekly benchmark on plafrim: experimental setup

Table of Contents

Back to index.

See the Experimental setup

1. Test case description

We consider a stationary heterogeneous diffusion equation (or Darcy equation) in a 3D stratified medium \(\nabla \cdot (k\nabla u) = 1\). The equation is discretized using the Finite Element Method. The matrices and right hand side are generated with genfem.

This benchmark is a weak scaling test with 18, 36, 72 and 144 subdomains. Each subdomain consists of a \(30 \times 30 \times 30\) cube. In the Y-direction, conductivity alternates every 5 elements, with an Heterogeneity of 1000.

Subdomain illustration: 30x30x30 cube

Figure 1: One subdomain, a cube of edge 30, conductivity \(k=1\) in blue and \(k=1000\) in red

For the baton test case, the subdomains are put one against the other in the X-dimension.

Baton test case illustration

Figure 2: Baton test case, with \(6 \times 1 \times 1\) subdomains

For the cuboid test case, the subdomains are put so as to form the "most cubic" shape possible.

Cuboid test case illustration

Figure 3: Cuboid test case, with \(4 \times 3 \times 2\) subdomains

2. Baton test case, on the input matrix K

2.1. openblas

mpp_K_openblas.png

2.2. mkl

mpp_K_mkl.png

3. Baton test case, on the Schur complement matrix S

3.1. openblas

mpp_S_openblas.png

3.2. mkl

mpp_S_mkl.png

4. Experiments in command line form

4.1. Preliminaries

rm -rf build && mkdir build && cd build

4.2. Command (Example)

For 36 subdomains with AS/S preconditionners using openblas

# creating Makefile
guix shell --pure -D composyx slurm eigen armadillo scalapack zlib -- cmake .. -DCMAKE_BUILD_TYPE=Release -DCOMPOSYX_COMPILE_EXAMPLES=ON -DCOMPOSYX_COMPILE_TESTS=OFF -DCOMPOSYX_DEV_WARNINGS=OFF
# compiling binary target
guix shell --pure -D composyx slurm eigen armadillo scalapack zlib -- make composyx_driver_cg

cd src/examples/

cat <<EOF > input.in
n_subdomains: 36
max_iter: 500

# Working on the Schur complement
use_schur: 1

# Path to partitions
partitions: /home/gitlab-compose/baton_30_newformat
# Partiton type
partition_type: baton
# Type of preconditioner
precond: AS
# Pastix or Mumps
direct_solver: Pastix

tolerance: 1e-6
EOF

cat <<EOF > batch.batch
#!/bin/bash
#SBATCH --job-name=composyx
#SBATCH --nodes=1
#SBATCH --ntasks=36
#SBATCH --ntasks-per-node=36
#SBATCH --cpus-per-task=1
#SBATCH --time=00:10:00
#SBATCH --output=slurm.out
#SBATCH --error=slurm.out
#SBATCH --exclusive
#SBATCH --constraint bora
module load mpi/openmpi/4.1.1
export OMPI_MCA_pml='^ucx'

guix shell --pure --preserve='^SLURM|^OMPI' -D composyx slurm eigen armadillo scalapack zlib -- mpiexec -n \${SLURM_NPROCS} ./composyx_driver_cg input.in
EOF

sbatch batch.batch

For 36 subdomains with AS/S preconditionners using mkl, you should add to the guix environment:

--with-input=mumps-openmpi=mumps-mkl-openmpi --with-input=openblas=mkl

Output is in slurm.out, errors are in slurm.err.

5. Source

Run date:

sam. 21 sept. 2024 01:14:40 CEST

Commit:

f9e360e6651df44fccc2ad6fc2d1f8ff7d225330

Channels:

(list (channel
        (name 'guix-hpc-non-free)
        (url "https://gitlab.inria.fr/guix-hpc/guix-hpc-non-free.git")
        (branch "master")
        (commit
          "5c78c7262f902c807e66852d326076d183b36b9c"))
      (channel
        (name 'guix)
        (url "https://git.savannah.gnu.org/git/guix.git")
        (branch "master")
        (commit
          "2c54c2db410ebdda8cd71716315e4ea4d31befbd")
        (introduction
          (make-channel-introduction
            "9edb3f66fd807b096b48283debdcddccfea34bad"
            (openpgp-fingerprint
              "BBB0 2DDF 2CEA F6A8 0D1D  E643 A2A0 6DF2 A33A 54FA"))))
      (channel
        (name 'guix-hpc)
        (url "https://gitlab.inria.fr/guix-hpc/guix-hpc.git")
        (branch "master")
        (commit
          "ac0563ef7ec5c49a4a79e533dcb0d7409eb26fd4"))
      (channel
        (name 'guix-science-nonfree)
        (url "https://github.com/guix-science/guix-science-nonfree.git")
        (branch "master")
        (commit
          "3413215081da370823c98e79445d8cbfd37531db")
        (introduction
          (make-channel-introduction
            "58661b110325fd5d9b40e6f0177cc486a615817e"
            (openpgp-fingerprint
              "CA4F 8CF4 37D7 478F DA05  5FD4 4213 7701 1A37 8446"))))
      (channel
        (name 'guix-science)
        (url "https://github.com/guix-science/guix-science.git")
        (branch "master")
        (commit
          "3736b80cf569511cc0a13f87cb03411761290dd8")
        (introduction
          (make-channel-introduction
            "b1fe5aaff3ab48e798a4cce02f0212bc91f423dc"
            (openpgp-fingerprint
              "CA4F 8CF4 37D7 478F DA05  5FD4 4213 7701 1A37 8446"))))
      (channel
        (name 'guix-past)
        (url "https://gitlab.inria.fr/guix-hpc/guix-past")
        (branch "master")
        (commit
          "792d830606345bddf5712252a1ed4fd87ee55c42")
        (introduction
          (make-channel-introduction
            "0c119db2ea86a389769f4d2b9c6f5c41c027e336"
            (openpgp-fingerprint
              "3CE4 6455 8A84 FDC6 9DB4  0CFB 090B 1199 3D9A EBB5")))))

results.csv

NbDomains,preconditioner,use_schur,blas,DirectSolver,nIter,CoarseEigenSolve,DirectSolve,CoarsePcdSetup,IterativeSolve,LocalPcdSetup,SchurComputation,TotalTime,RunTime,GlobMatOrder,robin_weight,threadspertask
36,diag,0,openblas,Mumps,2907,0,0,0,6.10487,0.00109815,0,6.11731,6.6241,1038876,1.0,1
36,diag,0,mkl,Mumps,2907,0,0,0,6.20122,0.0010703,0,6.21354,6.77482,1038876,1.0,1
36,diag,1,openblas,Mumps,445,0,0.0511532,0,2.07254,0.000211748,1.37708,3.50242,4.02217,1038876,1.0,1
36,diag,1,mkl,Mumps,445,0,0.0408961,0,2.08148,0.000166859,1.2757,3.4026,4.04323,1038876,1.0,1
36,AS,0,openblas,Mumps,353,0,0,0,12.5081,0.0827824,0,12.5201,13.0658,1038876,1.0,1
36,AS,0,mkl,Mumps,353,0,0,0,11.8411,0.0838694,0,11.8555,12.4539,1038876,1.0,1
36,AS,1,openblas,Mumps,59,0,0.0507074,0,1.01046,0.0829487,1.39068,2.45522,2.96013,1038876,1.0,1
36,AS,1,mkl,Mumps,59,0,0.0400108,0,0.744918,0.0846294,1.2888,2.07827,3.46026,1038876,1.0,1

Date: 27/09/2024 at 12:18:24

Author: Concace Concace

Created: 2024-09-27 Fri 12:18

Validate