# AutoIcon

Available examples:
- **real-from-ideal**: Start with a test simulation (ltestcase=.TRUE.) to produce a first-guess file and an analysis file to start a second simulation. 
- **real-from-dwd-ana**: Start with a first-guess and analysis from DWD (relying on the local archive to do that.)

> **_NOTE:_**  The current implementation of **real-from-dwd-ana** relies on the local archives at LMU. The templates to provide the initial conditions have to be modified to run from a different local system.

With autosubmit installed, we can run these ICON workflows with **autosubmit** we need to do the following:


## 1. Create an experiment:

    ```bash
    autosubmit expid -min -repo  https://gitlab.physik.uni-muenchen.de/w2w/autoicon.git -d "Launching an ICON experiment with autosubmit."
    ```
    
    This will create a new experiment folder and will give us the corresponding experiment id.

## 2. Modify the **minimal.yml** configuration file

    If we go to the **conf** folder corresponding to our experiment, we will find a **minimal.yml** file there:
```yaml
CONFIG:
  AUTOSUBMIT_VERSION: "4.0.76"
  TOTALJOBS: 20
  MAXWAITINGJOBS: 20
DEFAULT:
  EXPID: "a00p"
  HPCARCH: "local"
  #hint: use %PROJDIR% to point to the project folder (where the project is cloned)
  CUSTOM_CONFIG: "%PROJDIR%/"
PROJECT:
  PROJECT_TYPE: "git"
  PROJECT_DESTINATION: "git_project"
GIT:
  PROJECT_ORIGIN: "https://gitlab.physik.uni-muenchen.de/w2w/autoicon.git"
  PROJECT_BRANCH: ""
  PROJECT_COMMIT: ""
  PROJECT_SUBMODULES: ""
  FETCH_SINGLE_BRANCH: True

```

We will modify this file as follows:
 - Point to the proper branch ("master" should be stable)
 - We will add a new variable to select which example we do want to run: `ICON_CASE: "real-from-ideal"`
 - We will modify **CUSTOM_CONFIG** to point to the desired configurations.
 - We will modify **HPCARCH** to point to our desired platform.
```yaml
# Select case between real-from-ideal, real-from-dwd-ana
ICON_CASE: "real-from-ideal"

CONFIG:
  AUTOSUBMIT_VERSION: "4.0.76"
  TOTALJOBS: 20
  MAXWAITINGJOBS: 20
DEFAULT:
  EXPID: "a00p"   # Will depend on the actual experiment id.
  HPCARCH: "LRZ"
  #hint: use %PROJDIR% to point to the project folder (where the project is cloned)
  CUSTOM_CONFIG:
    PRE:
      - "%PROJDIR%/conf/common"
      - "%PROJDIR%/conf/%ICON_CASE%"

PROJECT:
  PROJECT_TYPE: "git"
  PROJECT_DESTINATION: "git_project"
GIT:
  PROJECT_ORIGIN: "https://gitlab.physik.uni-muenchen.de/w2w/autoicon.git"
  PROJECT_BRANCH: "master"
  PROJECT_COMMIT: ""
  PROJECT_SUBMODULES: ""
  FETCH_SINGLE_BRANCH: True

```

## 3. Additional configuration

We will create an additional yaml file with some more configuration.
> **_NOTE:_**  It is important to add these things in a different file because **minimal.yml** is the first file to be read,
> and all the other configuration files overwrite its contents.
> If we try to replace a variable that also appears in the default configuration files (i.e. local_destination_folder) it won't have an effect.

The name of the file doesn't matter as long as it is a **.yml** file, for example **my_configuration.yaml**.

To this file we will add the following:
 - We will add a path in the local machine at which the output data will be transferred.
```yaml
data_management:
  # Where do we put the output files afterwards?
  local_destination_folder: /Path/to/output/folder
```
 - Some details about how to build icon.

 Because we are using spack to build icon. The default command that would be used to build icon would be:

```bash
spack install icon-nwp@version%compiler
```

> **_NOTE:_**  Since in the LMU example we are building everything from scratch, probably it is the best starting point to try to build icon with spack in a new platform.

However, in some systems it is necessary to link to specific libraries or use existing packages.
Some additional parameters need to be changed in the configuration to allow that.
Here we have two examples:
- At **LMU**:

```yaml
spack:
  compiler: "gcc@11.3.0" # desired compiler for spack
  root: "$SCRATCH/autoicon-spack" # path to a spack install, will be downloaded to if not present
  externals: "slurm"
  user_cache_path:  "$SCRATCH/autoicon-spackcache" # spack puts data here when bootstrapping, leave empty to use home folder
  user_config_path: "$SCRATCH/autoicon-spackconfig" # spack puts data here when bootstrapping, leave empty to use home folder
  disable_local_config: false # if true, spack installs into spack source dir
  upstreams: "/software/opt/focal/x86_64/spack/2023.02/spack/opt/spack"

icon:
  build_cmd: "icon-nwp@%ICON.VERSION%% %SPACK.COMPILER%+debug~mpichecks target=x86_64_v2 source=dkrz_https"
  version: ${ICON_VERSION}
```

- At **LRZ**:

```yaml
spack:
  init: "module load user_spack" # command to load spack environment, e.g. module load spack, use spack/setup-env.sh if empty
  compiler: "gcc@11.2.0" # desired compiler for spack
  root: "$PLATFORM_SCRATCH/autoicon-spack" # path to a spack install, will be downloaded to if not present
  externals: "slurm"
  user_cache_path:  "$PLATFORM_SCRATCH/autoicon-spackcache" # spack puts data here when bootstrapping, leave empty to use home folder
  user_config_path: "$PLATFORM_SCRATCH/autoicon-spackconfig" # spack puts data here when bootstrapping, leave empty to use home folder

icon:
  build_cmd: "icon-nwp@%ICON.VERSION%% %SPACK.COMPILER%~mpichecks source=dkrz_https ^openmpi/amct7nx"
  load_cmd: "icon-nwp@%ICON.VERSION%% %SPACK.COMPILER%~mpichecks source=dkrz_https"
  version: 2.6.5-nwp0
```


> **_NOTE:_** At LRZ, the compute nodes don't have git available by default.

To overcome this problem we can compile the model and setup the python environment in the login nodes.
To do that, we can overwrite in which platform these jobs will be running by adding the following content to the file **my_configuration.yml**.

```yaml
Jobs:
  BUILD_ICON:
    PLATFORM: LRZ_LOGIN
  BUILD_PYTHON_ENVIRONMENT:
    PLATFORM: LRZ_LOGIN

```

## 4. Platforms

The user needs to provide a platforms file. It can be done in different ways but the easiest is to put the yaml file in the configuration folder.
> **_NOTE:_** If we need to run in a login node (i.e. LRZ_LOGIN) we need to add this as a platform as well. Specifying type=ps . 

## 5. Github password-less access
In order to get icon-sources through autosubmit, the password-less through ssh keys should be configured for dkrz gitlab.

## 6. Create and run
With this changes in the configuration should be already possible to create the experiment and run it.