Create an executable

1) Create directory

First create yourself a directory under your home(!) (so it gets backup up) in which you want to create the executable.
I usually call mine:

     ~/gem/v_5.1.1/Abs/...

For example:

           mkdir -p ~/gem/v_5.1.1/Abs/My_Abs      # Set 'My_Abs' to whatever you like. But make sure to give every executable directory a different 'my_Abs' name!

2) Populate directory

Go into the executable directory you just created above. For example:

            cd ~/gem/v_5.1.1/Abs/My_Abs

a) Get the basename of abs-directory

Get the basename (last part of your directory path) of your abs-directory for later use:

             base_dir=$( basename $(pwd) )

'base_dir' should now be set to 'My_Abs'. You can verify it with:

            echo $base_dir

b) Clone the Git repository you want

Before cloning anything make sure the owner of the code you want to clone has commited all modifications! Otherwise you will not get them. Then you can clone the source code with, for example:

             git clone ~winger/gem/v_5.1.1/Abs/UQAM_latest/gem

c) Create working directories

First change into 'gem' directory that came with the clone:

             cd ~/gem/v_5.1.1/Abs/My_Abs/gem

So the object files, libraries and executables will not be under your home, create links for the directories 'build' and 'work'. Pick a place under which you want to keep the object files, libraries and executables. Preferably under the default(!) space of you professors, for example:

            work_space=/project/def-professor/${USER}/GEM_WORK_DIR     # Set 'professor' to the CC name of your professor. You can find all of them under ~/projects.

Create the directories and the links (using ${base_dir} from above) :

            mkdir -p ${work_space}/${base_dir}/build
            mkdir -p ${work_space}/${base_dir}/work
            ln -s ${work_space}/${base_dir}/build
            ln -s ${work_space}/${base_dir}/work

3) Create the executable

a) Acquire compiler

(Only needed once per window/terminal.)

            module add intel/2021.2.0 fftw-intel-2021

b) If you are not already in it, go into the 'gem' directory of your abs-directory:

            cd ~/gem/v_5.1.1/Abs/My_Abs/gem

c) Set the variable 'abs_dir' to the name of your executable directory, including the 'gem' directory (needed under e))

            abs_dir=$(pwd)

d) Change into 'build' directory:

            cd build

e) Create make files and dependencies

(Only needed once and whenever you add any new routines or add include files or module "use" lines to an existing routine.)

            cmake   -DCOMPILER=intel   -DCMAKE_VERBOSE_MAKEFILE=ON    -DWORK_PREFIX=${abs_dir}/work    ${abs_dir}/project

f) Compile and create executable

            make -j4 maingemdm

If the executable got created, you will find it under:

            ls -lh src/maingem/maingemdm

It should have close to 50 MB.

4) Modify source code

a) You can find the main GEM routines under .../gem/src/*/*, for example :

Most dynamic routines are under:

            .../gem/src/gemdyn/base

Most physics routines are under:

            .../gem/src/rpnphy/base

Most surface routines are under:

            .../gem/src/rpnphy/surface

Find and edit the routines you want to modify.

b) Recreate the executable

Once you modified the routine(s) you want you need to recreate the executable.

To do this you need to go back in the 'build' directory!

Only if you add any new routines or add include files or module "use" lines to an existing routine you first need to recreate the dependencies and make files by re-executing the full cmake command under point 3d) again - see above.

Otherwise, it is enough to just re-execute:

            make -j4 maingemdm

If you just want to compile a routine without creating the executable (to save time) you can just compile a routine, for example utils/sfclayer_mod.F90 with:

            cd .../build/src/rpnphy
            make utils/sfclayer_mod.F90.o

5) Use your executable

To use your own executable set 'GEM_ovbin' in your 'configexp.cfg' to the path of your executable directory, up until but excluding the 'gem' directory. For example:

            GEM_ovbin=~/gem/v_5.1.1/Abs/my_abs

Modify compiler flags

Compile one routine without optimization, '-O0'

• edit the file named CMakeLists.txt in the main directory where the subroutine is situated (for example, src/rpnphy/CMakeLists.txt if the subroutine is situated in the physics directory)
• copy the following example before the last line (add_library...) and modify it according to your needs:

set_source_files_properties(utils/sfclayer_mod.F90 PROPERTIES COMPILE_OPTIONS "-O0")

• make sure you use a recent version of cmake (at least 3.11) - on Narval we do
  
• you can verify the option was passed by looking at the file flags.make situated in the corresponding directory in the build directory, for example build/src/rpnphy/CMakeFiles/phy.dir/flags.make:

# Custom options: src/rpnphy/CMakeFiles/phy.dir/utils/sfclayer_mod.F90.o_OPTIONS = -O0

Then go back into your 'build' directory and compile with the usual:

    make maingemdm
or
    make -j4 maingemdm

When compiling this routine now you should see something like the following:
Building Fortran object src/rpnphy/CMakeFiles/phy.dir/utils/sfclayer_mod.F90.o
...  -O2 -g   -O0 -c ...

The second '-O0' should overwrite the first '-O2'.

Set 'fp-model' to 'precise' or 'strict'

In project/Linux-x86_64-intel.cmake change in line 24 '-fp-model source' to '-fp-model precise' (Vivian) resp. '-fp-model strict' (Ron).