GOAS

GOAS (Gem On A Stick)

Part 1

Create source code and working directories, download (clone) source code, and database.
Part 1 only needs to be done once!

Create source code directory

First create and change into a directory under your home under which you will keep the source code. For example:

    mkdir -p ~/gem/v_5.1.1/GOAS/my_version
    cd ~/gem/v_5.1.1/GOAS/my_version

Clone and checkout GOAS version

    (Old place: git clone https://gitlab.com/eccc/gem/gem.git)
    git clone https://github.com/ECCC-ASTD-MRD/gem.git
    cd gem
    git checkout 5.1.1

Create and change into new branch

    git checkout -b my_branch

From here on we will refer to this directory as the "gem" directory.

Create 'work' and 'build' directories

In the 'build' directory will be your:

• CMakeFiles
• module files
• object files
• executables

In the 'work' directory will be:

• copy of the excecutables
• model scripts
• example config files
• example database
• and you will execute your simulation from here

Since these directories can get large you must not create them under your home but under you data space.

    MY_WORK_DIR=/path/to/my/work/directory

After you created the work directory you need to create some symbolic links to this directory:

    mkdir -p ${MY_WORK_DIR}/build
    mkdir -p ${MY_WORK_DIR}/work
    ln -s ${MY_WORK_DIR}/build
    ln -s ${MY_WORK_DIR}/work

Download database

GOAS comes with it's own database (initial conditions, geophysical & climatological fields, driving data). You only need to download it if you want to run any of the GOAS test configurations. Before downloading it I suggest you create a directory for it outside your home and create a symbolic to this directory, called 'gem_dbase'. For example:

    mkdir ${MY_WORK_DIR}/work/gem_dbase
    ln -s ${MY_WORK_DIR}/work/gem_dbase

Then you can download it with:

    download-dbase.sh .

Part 2

Set compiler version and create Makefiles

Change into the 'gem/build' directory and make sure you did export our work directory - see above:

    export MY_WORK_DIR=/path/to/my/work/directory

Also, to make things easier further down, set MY_GEM_DIR to the name of your "gem" directory - see above (replace the '...' in the following command!):

    export MY_GEM_DIR=.../gem

Change into the 'gem/build' directory:

    cd ${MY_GEM_DIR}/build

Depending on which machine you work and which compiler or environment you want to use the commands differ a little. Here are some options:

At UQAM

• If you want to use the Intel compiler set:

      module load openmpi/2.1.5-intel-19.0-mkl
      module load utils/misc                                                            # Only on CentOS 7 servers! To get an up to date version of cmake
      cmake -DCOMPILER=intel -DCMAKE_VERBOSE_MAKEFILE=ON -DWORK_PREFIX=${MY_WORK_DIR} ${MY_GEM_DIR}/project

• If you want to use the GNU compiler (gcc/gfortran) set:

      module load openmpi/3.1.5-gcc-8.3
      cmake -DCOMPILER=gnu -DCMAKE_VERBOSE_MAKEFILE=ON -DWORK_PREFIX=${MY_WORK_DIR} ${MY_GEM_DIR}/project

On Beluga

      module load intel/2019.3
      module load openmpi
      module load cmake/3.16.3
      module load fftw-mpi
      cmake -DCOMPILER=intel -DCMAKE_VERBOSE_MAKEFILE=ON -DWORK_PREFIX=${MY_WORK_DIR} ${MY_GEM_DIR}/project

On Niagara

• using the native environment (recommended)

      module load intel/2019u4 openmpi/4.0.1 fftw/3.3.8 cmake
      cmake -DCOMPILER=intel -DCMAKE_VERBOSE_MAKEFILE=ON -DWORK_PREFIX=${MY_WORK_DIR} ${MY_GEM_DIR}/project

• using the common environment

      module load CCEnv
      module load nixpkgs/16.09
      module load intel/2019.3
      module load openmpi
      module load cmake/3.16.3
      module load fftw-mpi
      cmake -DCOMPILER=intel -DCMAKE_VERBOSE_MAKEFILE=ON -DWORK_PREFIX=${MY_WORK_DIR} ${MY_GEM_DIR}/project

Create executable

    make -j8 work      # When using the key '-jn', the make command will run 'n' makes in parallel. So in this case 8.

Note: Sometimes, when running make in parallel it does not compile the routines in the right order (especially after massive updates of the source code). In that case run the command without the key '-j8'.

Run GOAS

To run GOAS you need to change into the model work directory (replace the '...' in the following command!):

    cd ${MY_WORK_DIR}
    cd work-...                             # change into work directory, i.e. work-linux26-x86-64-intel-19.0.3.199

Here you will find several example config files under 'configurations'. If you want to run, for example, a global YinYang simulation execute:

    ./runprep -dircfg configurations/GY_cfgs
    ./runmod -dircfg configurations/GY_cfgs -ptopo 2x2x1

Changing the code

Before making any changes to the source code it is recommended to create a new branch, for example:

    git checkout -b b_5.1-u1-climate

Once you change the source code follow the steps above, starting from "Set compiler version and compile".