From 0f0504b87460f160b452f65a1a7e82aa26187c23 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Oriol=20Tint=C3=B3?= <oriol.tinto@lmu.de>
Date: Fri, 26 May 2023 16:57:08 +0200
Subject: [PATCH] Update README.md

---
 README.md | 66 +++++++++++++++++++++++++++++++++++++++++++------------
 1 file changed, 52 insertions(+), 14 deletions(-)

diff --git a/README.md b/README.md
index 78c0974..78500bd 100644
--- a/README.md
+++ b/README.md
@@ -45,8 +45,7 @@ GIT:
 We will modify this file as follows:
  - Point to the proper branch ("master" should be stable)
  - We will add a new variable to select which example we do want to run: `ICON_CASE: "real-from-ideal"`
- - We will modify **CUSTOM_CONFIG** to point to the desired configurations. 
- - We will add a path in the local machine at which the output data will be transferred.
+ - We will modify **CUSTOM_CONFIG** to point to the desired configurations.
  - We will modify **HPCARCH** to point to our desired platform.
 ```yaml
 # Select case between real-from-ideal, real-from-dwd-ana
@@ -75,15 +74,27 @@ GIT:
   PROJECT_SUBMODULES: ""
   FETCH_SINGLE_BRANCH: True
 
+```
+
+## 3. Additional configuration
+
+We will create an additional yaml file with some more configuration.
+> **_NOTE:_**  It is important to add these things in a different file because **minimal.yml** is the first file to be read,
+> and all the other configuration files overwrite its contents.
+> If we try to replace a variable that also appears in the default configuration files (i.e. local_destination_folder) it won't have an effect.
+
+The name of the file doesn't matter as long as it is a **.yml** file, for example **my_configuration.yaml**.
+
+To this file we will add the following:
+ - We will add a path in the local machine at which the output data will be transferred.
+```yaml
 data_management:
   # Where do we put the output files afterwards?
   local_destination_folder: /Path/to/output/folder
-
 ```
+ - Some details about how to build icon.
 
-## 3. Modify build command.
-
-We are using spack to build icon. The default command that would be used to build icon would be:
+ Because we are using spack to build icon. The default command that would be used to build icon would be:
 
 ```bash
 spack install icon-nwp@version%compiler
@@ -94,32 +105,59 @@ spack install icon-nwp@version%compiler
 However, in some systems it is necessary to link to specific libraries or use existing packages.
 Some additional parameters need to be changed in the configuration to allow that.
 Here we have two examples:
-- At **LMU** we need to compile openmpi with a specific list of fabrics, and we need to use the external slurm as well. To do that we provide this additional configuration file:
+- At **LMU**:
 
 ```yaml
 spack:
+  compiler: "gcc@11.3.0" # desired compiler for spack
+  root: "$SCRATCH/autoicon-spack" # path to a spack install, will be downloaded to if not present
   externals: "slurm"
+  user_cache_path:  "$SCRATCH/autoicon-spackcache" # spack puts data here when bootstrapping, leave empty to use home folder
+  user_config_path: "$SCRATCH/autoicon-spackconfig" # spack puts data here when bootstrapping, leave empty to use home folder
+  disable_local_config: false # if true, spack installs into spack source dir
+  upstreams: "/software/opt/focal/x86_64/spack/2023.02/spack/opt/spack"
+
 icon:
-  build_cmd: "icon-nwp@%ICON.VERSION%% %SPACK.COMPILER% ^openmpi+pmi+legacylaunchers schedulers=slurm fabrics=ucx ucx+dc+dm+ib_hw_tm+mlx5_dv+rc+rdmacm+thread_multiple+ud+verbs"
+  build_cmd: "icon-nwp@%ICON.VERSION%% %SPACK.COMPILER%+debug~mpichecks target=x86_64_v2 source=dkrz_https"
+  version: ${ICON_VERSION}
 ```
 
-- At **LRZ** we rely on the spack package already available in the system that allows us to reuse most of the packages. To do so we add the following configuration:
+- At **LRZ**:
 
 ```yaml
 spack:
   init: "module load user_spack" # command to load spack environment, e.g. module load spack, use spack/setup-env.sh if empty
-  root: "" # Setting this to an empty string to use the existing module.
-  compiler: gcc@11.2.0 # desired compiler for spack. Matching the one used for the existing packages in LRZ
+  compiler: "gcc@11.2.0" # desired compiler for spack
+  root: "$PLATFORM_SCRATCH/autoicon-spack" # path to a spack install, will be downloaded to if not present
+  externals: "slurm"
+  user_cache_path:  "$PLATFORM_SCRATCH/autoicon-spackcache" # spack puts data here when bootstrapping, leave empty to use home folder
+  user_config_path: "$PLATFORM_SCRATCH/autoicon-spackconfig" # spack puts data here when bootstrapping, leave empty to use home folder
 
 icon:
-  build_cmd: "icon-nwp@%ICON.VERSION%% %SPACK.COMPILER% ^openmpi/amct7nx"
+  build_cmd: "icon-nwp@%ICON.VERSION%% %SPACK.COMPILER%~mpichecks source=dkrz_https ^openmpi/amct7nx"
+  load_cmd: "icon-nwp@%ICON.VERSION%% %SPACK.COMPILER%~mpichecks source=dkrz_https"
+  version: 2.6.5-nwp0
+```
+
+
+> **_NOTE:_** At LRZ, the compute nodes don't have git available by default.
+
+To overcome this problem we can compile the model and setup the python environment in the login nodes.
+To do that, we can overwrite in which platform these jobs will be running by adding the following content to the file **my_configuration.yml**.
+
+```yaml
+Jobs:
+  BUILD_ICON:
+    PLATFORM: LRZ_LOGIN
+  BUILD_PYTHON_ENVIRONMENT:
+    PLATFORM: LRZ_LOGIN
+
 ```
 
 ## 4. Platforms
 
 The user needs to provide a platforms file. It can be done in different ways but the easiest is to put the yaml file in the configuration folder.
-Because we wanted to execute some of the lightweight runs on the login nodes of the remote platform, it is necessary to define not only the desired platform but also an additional login.
-For example, if we use the platform **LMU**, we need to define **LMU_LOGIN** as well.
+> **_NOTE:_** If we need to run in a login node (i.e. LRZ_LOGIN) we need to add this as a platform as well. Specifying type=ps . 
 
 ## 5. Github password-less access
 In order to get icon-sources through autosubmit, the password-less through ssh keys should be configured for dkrz gitlab.
-- 
GitLab