Code wiki

GOAS (Gem On A Stick)

Part 1

Create source code and working directories, download (clone) source code, and database.
Part 1 only needs to be done once!

Create source code directory

First create and change into a directory under your home under which you will keep the source code. For example:

mkdir -p ~/gem/v_5.1.0/GOAS/UQAM_basic_mods
cd ~/gem/v_5.1.0/GOAS/UQAM_basic_mods

Clone and checkout latest GOAS version

    git clone https://gitlab.com/eccc/gem/gem.git
    cd gem
    git checkout -b b_5.1.0-rc6 5.1.0-rc6                   # rc6 latest as of 2021/02/10

Change into the directory 'gem':

cd gem

From here on we will refer to this directory as the "gem" directory.

Create work and 'build' directory

In this directory will be your:

object files and excecutables,
model scripts,
example config files,
example database,
and you will execute your simulation from here

Since this directory can get large you must not create it under your home but under you data space.

mkdir -p /path/to/my/work/directory
export MY_WORK_DIR=/path/to/my/work/directory

After you created the work directory you need to create some symbolic links to this directory:

    mkdir -p ${MY_WORK_DIR}
    ln -s ${MY_WORK_DIR} MY_WORK_DIR
    mkdir ${MY_WORK_DIR}/build
    ln -s ${MY_WORK_DIR}/build

Download database

GOAS comes with it's own database (initial conditions, geophysical & climatological fields, driving data). You only need to download it if you want to run any of the GOAS test configurations. Before downloading it I suggest you create a directory for it outside your home and create a symbolic to this directory, called 'gem_dbase'. For example:

mkdir ${MY_WORK_DIR}/gem_dbase
ln -s ${MY_WORK_DIR}/gem_dbase

Then you can download it with:

download-dbase.sh .

Part 2

Set compiler version and compile

Change into the 'gem' directory under which you downloaded your source code and make sure you did export our work directory:

export MY_WORK_DIR=/path/to/my/work/directory

(In case you forgot where this directory is, just check where your link 'MY_WORK_DIR' directory points to.)

Also, to make things easier further down, set MY_GEM_DIR to the name of your "gem" directory - see above (replace the '...' in the following command!):

export MY_GEM_DIR=.../gem

Depending on which machine you work and which compiler or environment you want to use the commands differ a little. Here are some options:

At UQAM

• If you want to use the Intel compiler set:

      module load openmpi/2.1.5-intel-19.0-mkl
      module load utils/misc    # to get an up to date version of cmake
      cmake -DCOMPILER=intel -DCMAKE_VERBOSE_MAKEFILE=ON -DWORK_PREFIX=${MY_WORK_DIR} ${MY_GEM_DIR}/project

• If you want to use the GNU compiler (gcc/gfortran) set:

module load openmpi/3.1.5-gcc-8.3
cmake -DCOMPILER=gnu -DCMAKE_VERBOSE_MAKEFILE=ON -DWORK_PREFIX=${MY_WORK_DIR} ${MY_GEM_DIR}/project

On Beluga

      module load intel/2019.3
      module load openmpi
      module load cmake/3.16.3
      module load fftw-mpi
      cmake -DCOMPILER=intel -DCMAKE_VERBOSE_MAKEFILE=ON -DWORK_PREFIX=${MY_WORK_DIR} ${MY_GEM_DIR}/project

On Niagara

• using the native environment (recommended)

module load intel/2019u4 openmpi/4.0.1 fftw/3.3.8 cmake
cmake -DCOMPILER=intel -DCMAKE_VERBOSE_MAKEFILE=ON -DWORK_PREFIX=${MY_WORK_DIR} ${MY_GEM_DIR}/project

• using the common environment

      module load CCEnv
      module load nixpkgs/16.09
      module load intel/2019.3
      module load openmpi
      module load cmake/3.16.3
      module load fftw-mpi
      cmake -DCOMPILER=intel -DCMAKE_VERBOSE_MAKEFILE=ON -DWORK_PREFIX=${MY_WORK_DIR} ${MY_GEM_DIR}/project

Create executable

make -j8 work # When using the key '-jn', the make command will run 'n' makes in parallel. So in this case 8.

Note: Sometimes, when running make in parallel it does not compile the routines in the right order (especially after massive updates of the source code). In that case run the command without the key '-j8'.

Run GOAS

To run GOAS you need to change into the model work directory (replace the '...' in the following command!):

cd ${MY_WORK_DIR}
cd work-... # change into work directory, i.e. work-linux26-x86-64-intel-19.0.3.199

Here you will find several example config files under 'configurations'. If you want to run, for example, a global YinYang simulation execute:

./runprep -dircfg configurations/GY_cfgs
./runmod -dircfg configurations/GY_cfgs -ptopo 2x2x1

Changing the code

Before making any changed to the source code it is recommended to create a new branch, for example:

git checkout -b b_5.1-u1-climate

Once you change the source code follow the steps above, starting from "Set compiler version and compile".