# Autosubmit-Icon-Today

Available examples:
- **real-from-ideal**: Start with a test simulation (ltestcase=.TRUE.) to produce a first-guess file and an analysis file to start a second simulation. 
- **real-from-dwd-ana**: Start with a first-guess and analysis from DWD (relying on the local archive to do that.)

> **_NOTE:_**  The current implementation of **real-from-dwd-ana** relies on the local archives at LMU. The templates to provide the initial conditions have to be modified to run from a different local system.

With autosubmit installed, we can run these ICON workflows with **autosubmit** we need to do the following:


## 1. Create an experiment:

    ```bash
    autosubmit expid -min -repo  https://gitlab.physik.uni-muenchen.de/w2w/autosubmit-icon-today.git -d "Launching an ICON experiment with autosubmit."
    ```
    
    This will create a new experiment folder and will give us the corresponding experiment id.

## 2. Modify the **minimal.yml** configuration file

    If we go to the **conf** folder corresponding to our experiment, we will find a **minimal.yml** file there:
```yaml
CONFIG:
  AUTOSUBMIT_VERSION: "4.0.76"
  TOTALJOBS: 20
  MAXWAITINGJOBS: 20
DEFAULT:
  EXPID: "a00p"
  HPCARCH: "local"
  #hint: use %PROJDIR% to point to the project folder (where the project is cloned)
  CUSTOM_CONFIG: "%PROJDIR%/"
PROJECT:
  PROJECT_TYPE: "git"
  PROJECT_DESTINATION: "git_project"
GIT:
  PROJECT_ORIGIN: "https://gitlab.physik.uni-muenchen.de/w2w/autosubmit-icon-today.git"
  PROJECT_BRANCH: ""
  PROJECT_COMMIT: ""
  PROJECT_SUBMODULES: ""
  FETCH_SINGLE_BRANCH: True

```

We will modify this file as follows:
 - Point to the proper branch ("master" should be stable)
 - We will add a new variable to select which example we do want to run: `ICON_CASE: "real-from-ideal"`
 - We will modify **CUSTOM_CONFIG** to point to the desired configurations. 
 - We will add a path in the local machine at which the output data will be transferred.
 - We will modify **HPCARCH** to point to our desired platform.
```yaml
# Select case between real-from-ideal, real-from-dwd-ana
ICON_CASE: "real-from-ideal"

CONFIG:
  AUTOSUBMIT_VERSION: "4.0.76"
  TOTALJOBS: 20
  MAXWAITINGJOBS: 20
DEFAULT:
  EXPID: "a00p"   # Will depend on the actual experiment id.
  HPCARCH: "LRZ"
  #hint: use %PROJDIR% to point to the project folder (where the project is cloned)
  CUSTOM_CONFIG:
    PRE:
      - "%PROJDIR%/conf/common"
      - "%PROJDIR%/conf/%ICON_CASE%"

PROJECT:
  PROJECT_TYPE: "git"
  PROJECT_DESTINATION: "git_project"
GIT:
  PROJECT_ORIGIN: "https://gitlab.physik.uni-muenchen.de/w2w/autosubmit-icon-today.git"
  PROJECT_BRANCH: "master"
  PROJECT_COMMIT: ""
  PROJECT_SUBMODULES: ""
  FETCH_SINGLE_BRANCH: True

data_management:
  # Where do we put the output files afterwards?
  local_destination_folder: /Path/to/output/folder

```

## 3. Modify build command.

We are using spack to build icon. The default command that would be used to build icon would be:

```bash
spack install icon-nwp@version%compiler
```

> **_NOTE:_**  Since in the LMU example we are building everything from scratch, probably it is the best starting point to try to build icon with spack in a new platform.

However, in some systems it is necessary to link to specific libraries or use existing packages.
Some additional parameters need to be changed in the configuration to allow that.
Here we have two examples:
- At **LMU** we need to compile openmpi with a specific list of fabrics, and we need to use the external slurm as well. To do that we provide this additional configuration file:

```yaml
spack:
  externals: "slurm"
icon:
  build_cmd: "icon-nwp@%ICON.VERSION%% %SPACK.COMPILER% ^openmpi+pmi+legacylaunchers schedulers=slurm fabrics=ucx ucx+dc+dm+ib_hw_tm+mlx5_dv+rc+rdmacm+thread_multiple+ud+verbs"
```

- At **LRZ** we rely on the spack package already available in the system that allows us to reuse most of the packages. To do so we add the following configuration:

```yaml
spack:
  init: "module load user_spack" # command to load spack environment, e.g. module load spack, use spack/setup-env.sh if empty
  root: "" # Setting this to an empty string to use the existing module.
  compiler: gcc@11.2.0 # desired compiler for spack. Matching the one used for the existing packages in LRZ

icon:
  build_cmd: "icon-nwp@%ICON.VERSION%% %SPACK.COMPILER% ^openmpi/amct7nx"
```

## 4. Platforms

The user needs to provide a platforms file. It can be done in different ways but the easiest is to put the yaml file in the configuration folder.
Because we wanted to execute some of the lightweight runs on the login nodes of the remote platform, it is necessary to define not only the desired platform but also an additional login.
For example, if we use the platform **LMU**, we need to define **LMU_LOGIN** as well.

## 5. Github password-less access
In order to get icon-sources through autosubmit, the password-less through ssh keys should be configured for dkrz gitlab. 

## 6. Create and run
With this changes in the configuration should be already possible to create the experiment and run it.