Some tweaks are necessary to work on LRZ, readme is outdated,

I am trying to setup ICON-NWP-PSP on the LRZ cluster, but many tweaks were necessary to build ICON. I will take a note here for some help to fix the issues.

1. we should create a new yml file in conf rather than putting everything minimal.yml
2. When compiling on LRZ, use we should use the login node the computing node of LRZ does not have general commands like git.
3. we need to ask a grant to access the dkrz-mirror group in the LRZ gitlab.
4. change the source repository of probtest coded in ~/spack_repo/packages/icon-nwp/package.py to https://github.com/MeteoSwiss/probtest. This might be specific when using the mirrored repositories in gitlab.lrz.de, but I am not sure.
5. Spack compilation seems successfully building ICON, but that cannot find the compiled package with ^openmpi/amct7nx option. Without this option, spack can find the compiled icon.
6. BUILD_PYTHON_ENVIRONMENT job fails with the error in installing cartopy,

fatal error: 'geos_c.h' file not found
  #include "geos_c.h"
           ^~~~~~~~~~

Further points will be added

geos_c is successfully built, but cartopy requires the static library

ra98sap@kcs-login:~/scratch/lxcusers/ra98sap/a003/LOG_a003> spack find --paths geos
==> 2 installed packages
-- linux-sles15-haswell / gcc@11.2.0 ----------------------------
geos@3.9.1  /dss/dsshome1/lrz/sys/spack/release/22.2.1/opt/haswell/geos/3.9.1-gcc-lfy4z7b

-- linux-sles15-skylake_avx512 / gcc@11.2.0 ---------------------
geos@3.9.1  /dss/dsshome1/lrz/sys/spack/release/22.2.1/opt/skylake_avx512/geos/3.9.1-gcc-ygffc7x


ra98sap@kcs-login:~/scratch/lxcusers/ra98sap/a003/LOG_a003> ls /dss/dsshome1/lrz/sys/spack/release/22.2.1/opt/haswell/geos/3.9.1-gcc-lfy4z7b/lib
cmake         libgeos_c.so.1       libgeos.so        pkgconfig
libgeos_c.so  libgeos_c.so.1.14.2  libgeos.so.3.9.1

• 1. we should create a new yml file in conf rather than putting everything minimal.yml
• 2. When compiling on LRZ, use we should use the login node the computing node of LRZ does not have general commands like git.
• 3. we need to ask a grant to access the dkrz-mirror group in the LRZ gitlab.
• 4. change the source repository of probtest coded in ~/spack_repo/packages/icon-nwp/package.py to https://github.com/MeteoSwiss/probtest. This might be specific when using the mirrored repositories in gitlab.lrz.de, but I am not sure.

We decided to switch to only use dkrz instead of the mirrors, we can rely on access tokens

• 5. Spack compilation seems successfully building ICON, but that cannot find the compiled package with ^openmpi/amct7nx option. Without this option, spack can find the compiled icon.

I added a new parameter that can be provided through the configuration files: load_cmd

In LRZ we can use the same command as build_cmd but without the dependency with the hash.

• 6. BUILD_PYTHON_ENVIRONMENT job fails with the error in installing cartopy,

enstools does not require cartopy as a strong dependency anymore

Thanks for the update! I tried the commit ce7918b9 and it worked on LRZ. What I only did manually is to add load_cmd in conf/my_conf.yml (not minimal.yml!) that has the same command without the openmpi specification.

icon:
  build_cmd: "icon-nwp@%ICON.VERSION%% %SPACK.COMPILER%~mpichecks source=dkrz_https ^openmpi/amct7nx"
  load_cmd: "icon-nwp@%ICON.VERSION%% %SPACK.COMPILER%~mpichecks source=dkrz_https"
  version: psp

• command spack install dwd-icon-tools % gcc@11.2.0
• spack install dwd-icon-tools % gcc@11.2.0 ==> Error: dwd-icon-tools version '2.5.2' has extra arguments: 'branch' Valid arguments for a url fetcher are: 'url', 'md5', 'sha1', 'sha224', 'sha256', 'sha384', 'sha512', and 'checksum'

After a lot of trials I found the way to successfully run ICON on LRZ. These are crucial points.

• Working only with use_spack/22.2.1 and gcc@11.2.0
• ecCodes command-line tools are optional >= spack 23.0. Remove "+tools" option from the dependency of the spack packages e.g. icon-nwp, dwd-icon-tools
• ECRad is using GNU extention. -std=gnu is necessary to use ECRad.
• with the intel compiler, The compile option "-check bounds" is crucial to avoid segmentation fault, but ICOn becomes extremely slow with the option, means practically we cannot use intel.

For someone wants to use ICON+ecrad later, just come to me or compare the main and D2_takumi branch to see what I have changed.

closed

Hi @Takumi.Matsunobu , we were happy enough using GNU compilers in LRZ, right? If it works fast enough I wouldn't spend too much time trying to figure out what is happening with intel but maybe I can let the LRZ support people know that we didn't manage to get everything working with the spack 23 thing.

Hi @Oriol.Tinto, yes, ICON using GNU with spack 22 in LRZ is running fast enough. So we can get rid of the intel compiler and spack 23 for the time being. Thinking of the time left for the KCS cluster (it won't be long, right?), I think we don't need to ask the LRZ about this issue.