Create an executable

1) Create directory

First create yourself a directory under your home(!) (so it gets backup up) in which you want to create the executable.
I usually call mine:

     ~/gem/v_5.1.1/Abs/...

For example:

           mkdir -p ~/gem/v_5.1.1/Abs/My_Abs      # Set 'My_Abs' to whatever you like. But make sure to give every executable directory a different 'my_Abs' name!

2) Populate directory

Go into the executable directory you just created above. For example:

            cd ~/gem/v_5.1.1/Abs/My_Abs

a) Get the basename of abs-directory

Get the basename (last part of your directory path) of your abs-directory for later use:

             base_dir=$( basename $(pwd) )

'base_dir' should now be set to 'My_Abs'. You can verify it with:

            echo $base_dir

b) Clone the Git repository you want

For example:

             git clone ~winger/gem/v_5.1.1/Abs/UQAM_latest/gem

then change into 'gem' directory that came with the clone:

             cd gem

c) Create working directories

So the object files, libraries and executables will not be under your home, create links for the directories 'build' and 'work'. Pick a place under which you want to keep the object files, libraries and executables. Preferably under the default(!) space of you professors, for example:

            work_space=/project/def-professor/${USER}/GEM_WORK_DIR     # Set 'professor' to the CC name of your professor. You can find all of them under ~/projects.

Create the directories and the links (using ${base_dir} from above) :

            mkdir -p ${work_space}/${base_dir}/build
            mkdir -p ${work_space}/${base_dir}/work
            ln -s ${work_space}/${base_dir}/build
            ln -s ${work_space}/${base_dir}/work

3) Create the executable

a) Acquire compiler

(Only needed once per window/terminal.)

            module add intel/2021.2.0 fftw-intel-2021

b) If you are not already in it, go into the 'gem' directory of your abs-directory:

            cd ~/gem/v_5.1.1/Abs/My_Abs/gem

c) Set the variable 'abs_dir' to the name of your executable directory, including the 'gem' directory (needed under e))

            abs_dir=$(pwd)

d) Change into 'build' directory:

            cd build

e) Create make files and dependencies

(Only need once at the beginning and whenever you add any new routines or add include files or module "use" lines to an existing routine.)

            cmake   -DCOMPILER=intel   -DCMAKE_VERBOSE_MAKEFILE=ON    -DWORK_PREFIX=${abs_dir}/work    ${abs_dir}/project

or add a flag to tell the compiler to compile P3 for GEM and not for WRF, although GEM should be the default:

            cmake   -DCMAKE_Fortran_FLAGS_INIT=-DECCCGEM  -DCOMPILER=intel   -DCMAKE_VERBOSE_MAKEFILE=ON    -DWORK_PREFIX=${abs_dir}/work    ${abs_dir}/project

f) Compile and create executable

            make -j4 maingemdm

The key '-j4' makes the command use 4 cores in parallel. But you can also run it on one core only with:

            make maingemdm

4) Modify source code

a) You can find the main GEM routines under .../gem/src/*/*, for example :

Most dynamic routines are under:

            .../gem/src/gemdyn/base

Most physics routines are under:

            .../gem/src/rpnphy/base

Most surface routines are under:

            .../gem/src/rpnphy/surface

Find and edit the routines you want to modify.

b) Recreate the executable

Once you modified the routine(s) you want you need to recreate the executable.

To do this you need to go back in the 'build' directory!

Only if you add any new routines or add include files or module "use" lines to an existing routine you first need to recreate the dependencies and make files by re-executing the full cmake command under point 3d) again - see above.

Otherwise, it is enough to just re-execute:

            make -j4 maingemdm

If you just want to compile a routine without creating the executable (to save time) you can just compile a routine, for example utils/sfclayer_mod.F90 with:

            cd .../build/src/rpnphy
            make utils/sfclayer_mod.F90.o

5) Use your executable

To use your own executable set 'GEM_ovbin' in your 'configexp.cfg' to the path of your executable directory, up until but excluding the 'gem' directory. For example:

            GEM_ovbin=~/gem/v_5.1.1/Abs/my_abs

Modify compiler flags

Compile one routine without optimization, '-O0'

• edit the file named CMakeLists.txt in the main directory where the subroutine is situated (for example, src/rpnphy/CMakeLists.txt if the subroutine is situated in the physics directory)
• copy the following example before the last line (add_library...) and modify it according to your needs:

set_source_files_properties(utils/sfclayer_mod.F90 PROPERTIES COMPILE_OPTIONS "-O0")

• make sure you use a recent version of cmake (at least 3.11) - on Narval we do
  
• you can verify the option was passed by looking at the file flags.make situated in the corresponding directory in the build directory, for example build/src/rpnphy/CMakeFiles/phy.dir/flags.make:

# Custom options: src/rpnphy/CMakeFiles/phy.dir/utils/sfclayer_mod.F90.o_OPTIONS = -O0

Then compile with the usual:

    make maingemdm
or
    make -j4 maingemdm

When compiling this routine now you should see something like the following:
Building Fortran object src/rpnphy/CMakeFiles/phy.dir/utils/sfclayer_mod.F90.o
...  -O2 -g   -O0 -c ...

The second '-O0' should overwrite the first '-O2'.

Set 'fp-model' to 'precise' or 'strict'

In project/Linux-x86_64-intel.cmake change in line 24 '-fp-model source' to '-fp-model precise' (Vivian) resp. '-fp-model strict' (Ron).