\( \newcommand{\blu}[1]{{\color{blue}#1}} \newcommand{\red}[1]{{\color{red}#1}} \newcommand{\grn}[1]{{\color{green!50!black}#1}} \newcommand{\local}{_i} \newcommand{\inv}{^{-1}} % Index for interface and interior \newcommand{\G}{\Gamma} \newcommand{\Gi}{\Gamma_i} \newcommand{\I}{{\cal I}} \newcommand{\Ii}{\I_i} % Matrix A \newcommand{\A}{{\cal A}} \newcommand{\Ai}{\A\local} \newcommand{\Aj}{\A_j} \newcommand{\Aib}{\bar{\A}\local} \newcommand{\AII}{\A_{\I\I}} \newcommand{\AIG}{\A_{\I\G}} \newcommand{\AGI}{\A_{\G\I}} \newcommand{\AGG}{\A_{\G\G}} \newcommand{\AIiIi}{\A_{\Ii\Ii}} \newcommand{\AIiGi}{\A_{\Ii\Gi}} \newcommand{\AGiIi}{\A_{\Gi\Ii}} \newcommand{\AGiGi}{\A_{\Gi\Gi}} \newcommand{\AGiGiw} {{\ensuremath{\A_{\Gi\Gi}^w}}} \newcommand{\AIIi}{\AII\local} \newcommand{\AIGi}{\AIG\local} \newcommand{\AGIi}{\AGI\local} \newcommand{\AGGi}{\AGG\local} \newcommand{\Ab}{\bar{\A}} \newcommand{\Ah}{{\widehat{\A}}} \newcommand{\Aih}{{\Ah\local}} \newcommand{\At}{{\widetilde{\A}}} \newcommand{\Ait}{{\At\local}} \newcommand{\Ao}{\A_0} \newcommand{\Aot}{\At_0} \newcommand{\Aob}{\Ab_0} \newcommand{\AiNN}{\Ai^{(NN)}{}} \newcommand{\AitNN}{\Ait^{(NN)}{}} \newcommand{\AiAS}{\Ai^{(AS)}{}} \newcommand{\AitAS}{\Ait^{(AS)}{}} % Matrix S \renewcommand{\S}{{\cal S}} \newcommand{\Si}{\S\local} \newcommand{\Sb}{\bar{\S}} \newcommand{\Sib}{\Sb\local} \newcommand{\Sh}{{\widehat{\S}}} \newcommand{\Sih}{{\Sh\local}} \newcommand{\St}{{\widetilde{\S}}} \newcommand{\Sit}{{\St\local}} \newcommand{\So}{\S_0} \newcommand{\Soi}{\S_0^i} \newcommand{\Sot}{\St_0} \newcommand{\Sob}{\Sb_0} \newcommand{\SiNN}{\Si^{(NN)}{}} \newcommand{\SitNN}{\Sit^{(NN)}{}} \newcommand{\SiAS}{\Si^{(AS)}{}} \newcommand{\SitAS}{\Sit^{(AS)}{}} % Matrix K \newcommand{\K}{{\cal K}} \newcommand{\Ki}{\K\local} \newcommand{\Kb}{\bar{\K}} \newcommand{\Kib}{\Kb\local} \newcommand{\Kh}{{\widehat{\K}}} \newcommand{\Kih}{{\Kh\local}} \newcommand{\Kt}{{\widetilde{\K}}} \newcommand{\Kit}{{\Kt\local}} \newcommand{\Ko}{\K_0} \newcommand{\Kot}{\Kt_0} \newcommand{\Kob}{\Kb_0} \newcommand{\KiNN}{\Ki^{(NN)}{}} \newcommand{\KitNN}{\Kit^{(NN)}{}} \newcommand{\KiAS}{\Ki^{(AS)}{}} \newcommand{\KitAS}{\Kit^{(AS)}{}} \newcommand{\KII}{\K_{\I\I}} \newcommand{\KIG}{\K_{\I\G}} \newcommand{\KGI}{\K_{\G\I}} \newcommand{\KGG}{\K_{\G\G}} \newcommand{\KIiIi}{\K_{\Ii\Ii}} \newcommand{\KIiGi}{\K_{\Ii\Gi}} \newcommand{\KGiIi}{\K_{\Gi\Ii}} \newcommand{\KGiGi}{\K_{\Gi\Gi}} \newcommand{\KIIi}{\KII\local} \newcommand{\KIGi}{\KIG\local} \newcommand{\KGIi}{\KGI\local} \newcommand{\KGGi}{\KGG\local} \newcommand{\KGiGiw} {{\ensuremath{\K_{\Gi\Gi}^w}}} % Matrix B \newcommand{\B}{{\cal B}} \newcommand{\Bi}{\B\local} \newcommand{\Bib}{\widehat{\B}\local} \newcommand{\Bob}{\widehat{\B}_0} % Matrix C \newcommand{\C}{{\cal C}} % Matrix T \newcommand{\T}{{\cal T}} \newcommand{\Ti}{{\T\local}} % Vectors \newcommand{\uI}{u_\I} \newcommand{\uG}{u_\G} \newcommand{\xI}{x_\I} \newcommand{\xG}{x_\G} \newcommand{\bI}{b_\I} \newcommand{\bG}{b_\G} \newcommand{\fI}{f_\I} \newcommand{\fG}{f_\G} \newcommand{\ftG}{\widetilde f_\G} \newcommand{\ftGi}{\widetilde f_\Gi^{(i)}} \newcommand{\ftGj}{\widetilde f_\Gj^{(j)}} \)

Composyx Composyx weekly benchmark on plafrim: experimental setup

Table of Contents

Back to index.

See the Experimental setup

1. Test case description

We consider a stationary heterogeneous diffusion equation (or Darcy equation) in a 3D stratified medium \(\nabla \cdot (k\nabla u) = 1\). The equation is discretized using the Finite Element Method. The matrices and right hand side are generated with genfem.

This benchmark is a weak scaling test with 18, 36, 72 and 144 subdomains. Each subdomain consists of a \(30 \times 30 \times 30\) cube. In the Y-direction, conductivity alternates every 5 elements, with an Heterogeneity of 1000.

Subdomain illustration: 30x30x30 cube

Figure 1: One subdomain, a cube of edge 30, conductivity \(k=1\) in blue and \(k=1000\) in red

For the baton test case, the subdomains are put one against the other in the X-dimension.

Baton test case illustration

Figure 2: Baton test case, with \(6 \times 1 \times 1\) subdomains

For the cuboid test case, the subdomains are put so as to form the "most cubic" shape possible.

Cuboid test case illustration

Figure 3: Cuboid test case, with \(4 \times 3 \times 2\) subdomains

2. Baton test case, on the input matrix K

2.1. openblas

mpp_K_openblas.png

2.2. mkl

mpp_K_mkl.png

3. Baton test case, on the Schur complement matrix S

3.1. openblas

mpp_S_openblas.png

3.2. mkl

mpp_S_mkl.png

4. Experiments in command line form

4.1. Preliminaries

rm -rf build && mkdir build && cd build

4.2. Command (Example)

For 36 subdomains with AS/S preconditionners using openblas 

# creating Makefile
guix shell --pure -D composyx slurm eigen armadillo scalapack zlib -- cmake .. -DCMAKE_BUILD_TYPE=Release -DCOMPOSYX_COMPILE_EXAMPLES=ON -DCOMPOSYX_COMPILE_TESTS=OFF -DCOMPOSYX_DEV_WARNINGS=OFF
# compiling binary target
guix shell --pure -D composyx slurm eigen armadillo scalapack zlib -- make composyx_driver_cg

cd src/examples/

cat <<EOF > input.in
n_subdomains: 36
max_iter: 500

# Working on the Schur complement
use_schur: 1

# Path to partitions
partitions: /home/gitlab-compose/baton_30_newformat
# Partiton type
partition_type: baton
# Type of preconditioner
precond: AS
# Pastix or Mumps
direct_solver: Pastix

tolerance: 1e-6
EOF

cat <<EOF > batch.batch
#!/bin/bash
#SBATCH --job-name=composyx
#SBATCH --nodes=1
#SBATCH --ntasks=36
#SBATCH --ntasks-per-node=36
#SBATCH --cpus-per-task=1
#SBATCH --time=00:10:00
#SBATCH --output=slurm.out
#SBATCH --error=slurm.out
#SBATCH --exclusive
#SBATCH --constraint bora
module load mpi/openmpi/4.1.1
export OMPI_MCA_pml='^ucx'

guix shell --pure --preserve='^SLURM|^OMPI' -D composyx slurm eigen armadillo scalapack zlib -- mpiexec -n \${SLURM_NPROCS} ./composyx_driver_cg input.in
EOF

sbatch batch.batch

For 36 subdomains with AS/S preconditionners using mkl, you should add to the guix environment: 

--with-input=mumps-openmpi=mumps-mkl-openmpi --with-input=openblas=mkl

Output is in slurm.out, errors are in slurm.err.

5. Source

Run date: 

sam. 01 févr. 2025 01:28:05 CET

Commit: 

7b9645ca8ee60930052d30b409860f0bf0d5d326

Channels: 

(list (channel
        (name 'guix-hpc-non-free)
        (url "https://gitlab.inria.fr/guix-hpc/guix-hpc-non-free.git")
        (branch "master")
        (commit
          "c546e8121b4d8a715dcb6cd743680dd7eee30b0e"))
      (channel
        (name 'guix)
        (url "https://git.savannah.gnu.org/git/guix.git")
        (branch "master")
        (commit
          "d48da2d21610f9cf5f76cd846703b12beedb1fd5")
        (introduction
          (make-channel-introduction
            "9edb3f66fd807b096b48283debdcddccfea34bad"
            (openpgp-fingerprint
              "BBB0 2DDF 2CEA F6A8 0D1D  E643 A2A0 6DF2 A33A 54FA"))))
      (channel
        (name 'guix-hpc)
        (url "https://gitlab.inria.fr/guix-hpc/guix-hpc.git")
        (branch "master")
        (commit
          "f9e48c009bee31b70188d2d1e15017f5b145f4d7"))
      (channel
        (name 'guix-science-nonfree)
        (url "https://codeberg.org/guix-science/guix-science-nonfree.git")
        (branch "master")
        (commit
          "de7cda4027d619bddcecf6dd68be23b040547bc7")
        (introduction
          (make-channel-introduction
            "58661b110325fd5d9b40e6f0177cc486a615817e"
            (openpgp-fingerprint
              "CA4F 8CF4 37D7 478F DA05  5FD4 4213 7701 1A37 8446"))))
      (channel
        (name 'guix-science)
        (url "https://codeberg.org/guix-science/guix-science.git")
        (branch "master")
        (commit
          "31387efc4d90362e79251953fd1a6554dc262cda")
        (introduction
          (make-channel-introduction
            "b1fe5aaff3ab48e798a4cce02f0212bc91f423dc"
            (openpgp-fingerprint
              "CA4F 8CF4 37D7 478F DA05  5FD4 4213 7701 1A37 8446"))))
      (channel
        (name 'guix-past)
        (url "https://codeberg.org/guix-science/guix-past.git")
        (branch "master")
        (commit
          "70fc56e752ef6d5ff6e1e1a0997fa72e04337b24")
        (introduction
          (make-channel-introduction
            "0c119db2ea86a389769f4d2b9c6f5c41c027e336"
            (openpgp-fingerprint
              "3CE4 6455 8A84 FDC6 9DB4  0CFB 090B 1199 3D9A EBB5")))))

results.csv 

NbDomains,preconditioner,use_schur,blas,DirectSolver,nIter,CoarseEigenSolve,DirectSolve,CoarsePcdSetup,IterativeSolve,LocalPcdSetup,SchurComputation,TotalTime,RunTime,GlobMatOrder,robin_weight,threadspertask
36,0,0,openblas,Mumps,4758,0,0,0,9.2288,0,0,9.24139,9.7412,1038876,1.0,1
36,0,0,mkl,Mumps,4758,0,0,0,9.70333,0,0,9.71634,10.8621,1038876,1.0,1
,0,0,openblas,,,,,,,,,,,,1.0,1
,0,0,mkl,,,,,,,,,,,,1.0,1
,0,0,openblas,,,,,,,,,,,,1.0,1
,0,0,mkl,,,,,,,,,,,,1.0,1
36,0,1,openblas,Mumps,901,0,0.0535269,0,4.26495,0,1.36955,5.70542,6.23264,1038876,1.0,1
36,0,1,mkl,Mumps,902,0,0.0418861,0,4.19285,0,1.27385,5.52907,6.20652,1038876,1.0,1
,0,1,openblas,,,,,,,,,,,,1.0,1
,0,1,mkl,,,,,,,,,,,,1.0,1
,0,1,openblas,,,,,,,,,,,,1.0,1
,0,1,mkl,,,,,,,,,,,,1.0,1
36,diag,0,openblas,Mumps,2907,0,0,0,6.00592,0.0012825,0,6.01813,6.52453,1038876,1.0,1
36,diag,0,mkl,Mumps,2907,0,0,0,6.23727,0.00134135,0,6.2502,6.94421,1038876,1.0,1
,diag,0,openblas,,,,,,,,,,,,1.0,1
,diag,0,mkl,,,,,,,,,,,,1.0,1
,diag,0,openblas,,,,,,,,,,,,1.0,1
,diag,0,mkl,,,,,,,,,,,,1.0,1
36,diag,1,openblas,Mumps,445,0,0.0509057,0,2.08659,0.000195519,1.38352,3.53447,4.06798,1038876,1.0,1
36,diag,1,mkl,Mumps,445,0,0.040106,0,2.09015,0.000233198,1.28422,3.43413,4.06431,1038876,1.0,1
,diag,1,openblas,,,,,,,,,,,,1.0,1
,diag,1,mkl,,,,,,,,,,,,1.0,1
,diag,1,openblas,,,,,,,,,,,,1.0,1
,diag,1,mkl,,,,,,,,,,,,1.0,1
36,AS,0,openblas,Mumps,353,0,0,0,12.5016,0.0854029,0,12.5125,13.0095,1038876,1.0,1
36,AS,0,mkl,Mumps,353,0,0,0,11.7291,0.0801168,0,11.7432,12.6505,1038876,1.0,1
,AS,0,openblas,,,,,,,,,,,,1.0,1
,AS,0,mkl,,,,,,,,,,,,1.0,1
,AS,0,openblas,,,,,,,,,,,,1.0,1
,AS,0,mkl,,,,,,,,,,,,1.0,1
36,AS,1,openblas,Mumps,59,0,0.0507062,0,1.01665,0.0826249,1.3758,2.46376,2.98327,1038876,1.0,1
36,AS,1,mkl,Mumps,59,0,0.0413109,0,0.738242,0.0826676,1.27872,2.07905,2.99321,1038876,1.0,1
,AS,1,openblas,,,,,,,,,,,,1.0,1
,AS,1,mkl,,,,,,,,,,,,1.0,1
,AS,1,openblas,,,,,,,,,,,,1.0,1
,AS,1,mkl,,,,,,,,,,,,1.0,1
36,RR,0,openblas,Mumps,2033,0,0,0,67.1603,0.0414204,0,67.1704,67.6828,1038876,1.0,1
36,RR,0,mkl,Mumps,2028,0,0,0,62.5327,0.0401239,0,62.5432,63.1319,1038876,1.0,1
,RR,0,openblas,,,,,,,,,,,,1.0,1
,RR,0,mkl,,,,,,,,,,,,1.0,1
,RR,0,openblas,,,,,,,,,,,,1.0,1
,RR,0,mkl,,,,,,,,,,,,1.0,1
36,RR,1,openblas,Mumps,96,0,0.051397,0,1.54806,0.0530844,1.3786,2.99593,3.50848,1038876,1.0,1
36,RR,1,mkl,Mumps,97,0,0.0403021,0,1.06715,0.0538634,1.27437,2.40321,3.1318,1038876,1.0,1
,RR,1,openblas,,,,,,,,,,,,1.0,1
,RR,1,mkl,,,,,,,,,,,,1.0,1
,RR,1,openblas,,,,,,,,,,,,1.0,1
,RR,1,mkl,,,,,,,,,,,,1.0,1
36,ASG+,0,openblas,Mumps,34,7.38161,0,6.62405,12.9393,0.298139,0,12.952,13.4633,1038876,1.0,1
36,ASG+,0,mkl,Mumps,35,7.06848,0,6.47182,12.4959,0.298521,0,12.5078,13.0798,1038876,1.0,1
,ASG+,0,openblas,,,,,,,,,,,,1.0,1
,ASG+,0,mkl,,,,,,,,,,,,1.0,1
,ASG+,0,openblas,,,,,,,,,,,,1.0,1
,ASG+,0,mkl,,,,,,,,,,,,1.0,1
36,ASG+,1,openblas,Mumps,4,0.483403,0.0508878,0.471113,0.899479,0.208995,1.37096,2.34082,2.8564,1038876,1.0,1
36,ASG+,1,mkl,Mumps,4,0.478471,0.0399464,0.46474,0.866589,0.212038,1.27423,2.20461,2.94918,1038876,1.0,1
,ASG+,1,openblas,,,,,,,,,,,,1.0,1
,ASG+,1,mkl,,,,,,,,,,,,1.0,1
,ASG+,1,openblas,,,,,,,,,,,,1.0,1
,ASG+,1,mkl,,,,,,,,,,,,1.0,1

Date: 04/02/2025 at 13:09:34

Author: Concace Concace

Created: 2025-02-04 Tue 13:09

Validate