AutoIcon
Available examples:
- real-from-ideal: Start with a test simulation (ltestcase=.TRUE.) to produce a first-guess file and an analysis file to start a second simulation.
- real-from-dwd-ana: Start with a first-guess and analysis from DWD (relying on the local archive to do that.)
NOTE: The current implementation of real-from-dwd-ana relies on the local archives at LMU. The templates to provide the initial conditions have to be modified to run from a different local system.
With autosubmit installed, we can run these ICON workflows with autosubmit we need to do the following:
1. Create an experiment:
```bash
autosubmit expid -min -repo https://gitlab.physik.uni-muenchen.de/w2w/autoicon.git -d "Launching an ICON experiment with autosubmit."
```
This will create a new experiment folder and will give us the corresponding experiment id.2. Modify the minimal.yml configuration file
If we go to the **conf** folder corresponding to our experiment, we will find a **minimal.yml** file there:CONFIG:
AUTOSUBMIT_VERSION: "4.0.76"
TOTALJOBS: 20
MAXWAITINGJOBS: 20
DEFAULT:
EXPID: "a00p"
HPCARCH: "local"
#hint: use %PROJDIR% to point to the project folder (where the project is cloned)
CUSTOM_CONFIG: "%PROJDIR%/"
PROJECT:
PROJECT_TYPE: "git"
PROJECT_DESTINATION: "git_project"
GIT:
PROJECT_ORIGIN: "https://gitlab.physik.uni-muenchen.de/w2w/autoicon.git"
PROJECT_BRANCH: ""
PROJECT_COMMIT: ""
PROJECT_SUBMODULES: ""
FETCH_SINGLE_BRANCH: True
We will modify this file as follows:
- Point to the proper branch ("master" should be stable)
- We will add a new variable to select which example we do want to run:
ICON_CASE: "real-from-ideal" - We will modify CUSTOM_CONFIG to point to the desired configurations.
- We will modify HPCARCH to point to our desired platform.
# Select case between real-from-ideal, real-from-dwd-ana
ICON_CASE: "real-from-ideal"
CONFIG:
AUTOSUBMIT_VERSION: "4.0.76"
TOTALJOBS: 20
MAXWAITINGJOBS: 20
DEFAULT:
EXPID: "a00p" # Will depend on the actual experiment id.
HPCARCH: "LRZ"
#hint: use %PROJDIR% to point to the project folder (where the project is cloned)
CUSTOM_CONFIG:
PRE:
- "%PROJDIR%/conf/common"
- "%PROJDIR%/conf/%ICON_CASE%"
PROJECT:
PROJECT_TYPE: "git"
PROJECT_DESTINATION: "git_project"
GIT:
PROJECT_ORIGIN: "https://gitlab.physik.uni-muenchen.de/w2w/autoicon.git"
PROJECT_BRANCH: "master"
PROJECT_COMMIT: ""
PROJECT_SUBMODULES: ""
FETCH_SINGLE_BRANCH: True
3. Additional configuration
We will create an additional yaml file with some more configuration.
NOTE: It is important to add these things in a different file because minimal.yml is the first file to be read, and all the other configuration files overwrite its contents. If we try to replace a variable that also appears in the default configuration files (i.e. local_destination_folder) it won't have an effect.
The name of the file doesn't matter as long as it is a .yml file, for example my_configuration.yaml.
To this file we will add the following:
- We will add a path in the local machine at which the output data will be transferred.
data_management:
# Where do we put the output files afterwards?
local_destination_folder: /Path/to/output/folder- Some details about how to build icon.
Because we are using spack to build icon. The default command that would be used to build icon would be:
spack install icon-nwp@version%compilerNOTE: Since in the LMU example we are building everything from scratch, probably it is the best starting point to try to build icon with spack in a new platform.
However, in some systems it is necessary to link to specific libraries or use existing packages. Some additional parameters need to be changed in the configuration to allow that. Here we have two examples:
- At LMU:
spack:
compiler: "gcc@11.3.0" # desired compiler for spack
root: "$SCRATCH/autoicon-spack" # path to a spack install, will be downloaded to if not present
externals: "slurm"
user_cache_path: "$SCRATCH/autoicon-spackcache" # spack puts data here when bootstrapping, leave empty to use home folder
user_config_path: "$SCRATCH/autoicon-spackconfig" # spack puts data here when bootstrapping, leave empty to use home folder
disable_local_config: false # if true, spack installs into spack source dir
upstreams: "/software/opt/focal/x86_64/spack/2023.02/spack/opt/spack"
icon:
build_cmd: "icon-nwp@%ICON.VERSION%% %SPACK.COMPILER%+debug~mpichecks target=x86_64_v2 source=dkrz_https"
version: ${ICON_VERSION}- At LRZ:
spack:
init: "module load user_spack" # command to load spack environment, e.g. module load spack, use spack/setup-env.sh if empty
compiler: "gcc@11.2.0" # desired compiler for spack
root: "$PLATFORM_SCRATCH/autoicon-spack" # path to a spack install, will be downloaded to if not present
externals: "slurm"
user_cache_path: "$PLATFORM_SCRATCH/autoicon-spackcache" # spack puts data here when bootstrapping, leave empty to use home folder
user_config_path: "$PLATFORM_SCRATCH/autoicon-spackconfig" # spack puts data here when bootstrapping, leave empty to use home folder
icon:
build_cmd: "icon-nwp@%ICON.VERSION%% %SPACK.COMPILER%~mpichecks source=dkrz_https ^openmpi/amct7nx"
load_cmd: "icon-nwp@%ICON.VERSION%% %SPACK.COMPILER%~mpichecks source=dkrz_https"
version: 2.6.5-nwp0NOTE: At LRZ, the compute nodes don't have git available by default.
To overcome this problem we can compile the model and setup the python environment in the login nodes. To do that, we can overwrite in which platform these jobs will be running by adding the following content to the file my_configuration.yml.
Jobs:
BUILD_ICON:
PLATFORM: LRZ_LOGIN
BUILD_PYTHON_ENVIRONMENT:
PLATFORM: LRZ_LOGIN
4. Platforms
The user needs to provide a platforms file. It can be done in different ways but the easiest is to put the yaml file in the configuration folder.
NOTE: If we need to run in a login node (i.e. LRZ_LOGIN) we need to add this as a platform as well. Specifying type=ps .
5. Github password-less access
In order to get icon-sources through autosubmit, the password-less through ssh keys should be configured for dkrz gitlab.
6. Create and run
With this changes in the configuration should be already possible to create the experiment and run it.