General information

Domain & resolution

First you should decide upon the final(!) (see "Cascading" below) domain and resolution you want to run. For this it is important to know that:

• the more grid points a grid has => the more cores are needed
• the higher the resolution => the shorter the model timestep => the longer the runtime of the simulation

Have a look at the following wiki about how to set up a model grid: Set up a model grid

Topology (MPI & OpenMP)

To speed up a simulation one can run it on more then one cpu. Or in other words, one can "run it in parallel". There are two ways to parallelize GEM simulations: MPI and OpenMP. Both get used at the same time by setting the parameter 'GEM_ptopo' from the file configexp.dot:

    GEM_ptopo=(MPI-x)x(MPI-y)x(OpenMP)      # See more explanations below

If more than 1 node will get used, the total number of cpus used should be a multiple of the number of cores of a compute node!

For Yin-Yang grids, the number of cores specified in 'GEM_ptopo' is the number used for each of the LAM grids. So the total number will be twice as high!

MPI (Message Passing Interface)

Using MPI to parallelize GEM is more efficient than using OpenMP. To parallelize the a simulation using MPI  the full model domain (including pilot region) gets divided into "tiles". Each MPI process will then take care of one tile. Meaning there will be as many MPI processes as there will be tiles.

To use MPI one needs to set the first two numbers of 'GEM_ptopo' to the number of tiles one would like to use in x- (MPI-x) and y-direction (MPI-y). For example, to use 2 tiles in x- and 2 tiles in y-direction set:

    GEM_ptopo=2x2x1

FYI, if the total number of points of the model grid is not exactly dividable by the number of MPI tiles, the first tile(s) will get more points and the other(s) less.

Basic rules to consider

• The minimum tile size is 23x23 points!
• The maximum tile size depends on the schemes used: The "memory heavier" a scheme, the smaller the tiles need to be. For example, 'P3' needs more memory than 'consun' and when adding 'TEB' the simulation will also need much more memory. In general, a maximum tile size of about 100x100 points should not get exceeded.
• Tiles should be as square as possible to minimize the information exchange over the tile borders.
• If tiles cannot be square and one has a choice, it is preferable to have more tiles in y- than in x-direction to help the solver.

OpenMP

In GEM the physics calculate one "slab" (row * levels) after the other. When using OpenMP, more than one slab will get treated a the same time. To parallelize the a simulation by using OpenMP one needs to set the last number of 'GEM_ptopo' to the number of OpenMP processes to get used.

It is NOT recommended to use more than two OpenMP processes. To use OpenMP=2 in the above example one would set:

    GEM_ptopo=2x2x2

Resolution → timestep

The relation between the horizontal resolution and the model timestep is linear. Here are some examples:

Resolution → timestep → radiation timestep

Calculating the increments on different fields depending on the radiation is rather costly. Therefore, the option exists to calculate the increments only every n seconds but still apply them at every timestep. The following parameters of the namelist 'physics_cfgs' can get set to a multiple 4 to 6 of the model timestep.

     'kntrad_S'      : Time between full radiation calculation (units D,H,M,S,P)
     'kntraduv_S' : Time between UV radiation calculation (units D,H,M,S,P)

Resolution → (non) hydrostatic

If the grid resolution is ~10 km or lower the model is hydrostatic and one should set 'Dynamics_hydro_L = .true.' in the namelist 'dyn_kernel'. For grid resolutions of ~4km and higher one should set 'Dynamics_hydro_L = .false.'

Model levels

The model levels on GEM are terrain following at the surface and pressure levels at the model top. The model levels are generally closer to each other at the surface and further apart higher up. The model is very sensitive to the location of these model levels. They must neither be too far apart nor too close and the transition from terrain following to pressure levels must be very smooth. There are ways to set up these model levels. Several sets of model levels have already been created and can get selected by setting 'CLIMAT_etaname' in the file configexp.cfg. The possible settings for this parameter can get found in the file:
      ~/modeles/GEMDM/*/bin/Climat_eta2preshyb

Vertical sponge & top piloting (LAM only)

Probably because waves get reflected back into the atmosphere from the model lid, a spurius wind jet forms at the equatorial stratopause. To remove this spurius wind jet, a so called "vertical sponge" is used at the model top to absorb these waves so they do not get reflected. For more information and how to set up the vertical sponge (Vspng) have a look at the following wiki: Diffusion & Sponges

In LAM model one also has the option of instead nesting the top model layers. Which is also called "top piloting".

When top piloting is used the vertical sponge should NOT get used!!!

Global grid → adjust global pressure

Due to the semi-lagrangian advection scheme, which does not exactly conserve the total mass of the atmosphere, the GEM model is "loosing air" with time. Therefore, the parameter 'Schm_psadj' from the namelist 'dyn_fisl' needs to get set to one of the following:

      !# * 0 -> No conservation of surface pressure (for LAM grids)
      !# * 1 -> Conservation of total air mass pressure (for global grids)
      !# * 2 -> Conservation of dry    air mass pressure (for global grids)

Cascadeing

When the resolution of the driving data is too large with respect to the model grid resolution, one or more simulations can get run with an intermediate resolution and a domain slightly larger than the next inner domain.

Spectral nudging

To constrain the large scales of a simulation spectral nudging can get used. To learn more about this and which parameters to set have a look at the following wiki page: Spectral nudging

Period

Start and end date of a simulation need to get set in the file configexp.cfg with the parameters:
    CLIMAT_startdate="YYYY MM DD hh"
    CLIMAT_enddate="YYYY MM DD hh"

Greenhouse gases / Scenario

Depending on the period (past, current, future) and for future simulation depending on the scenario simulated, a table containing the annual mean concentrations of the main 5 greenhouse gases (CO2, N2O, CH4, CFC-11, CFC-12) needs to get assigned to the parameter 'CLIMAT_ghg_list' (in file configexp.cfg). A selection of different lists can be found under: ${MODEL_DATA}/Greenhouse_gases

But one can also create their own list.

Initial conditions

At the beginning of every simulations several fields (atmospheric, surface and soil) need to get initialized. Especially for short simulations it is very important to use fields that best represent the starting state of a simulation. If such fields are not available one can also choose to spinup the simulation. Whereas atmospheric fields get into equilibrium very quickly, surface fields like snow on the ground might take up to at least a season. Soil fields, like soil temperature and soil moisture, can take several years to get into equilibrium, depending on the soil depth.

To get a list of fields that need to get initialized have a look at the following wiki: Initial conditions

Driving data (lower & lateral)

Lower boundary conditions (SST & SIC)

Since our(!!!) GEM version, CRCM-GEM, is not coupled to an ocean model, the sea surface temperature (SST) as well as the sea ice fraction (SIC) need to get prescribed. These two fields can get prescribed at every model timestep up to monthly fields. Most of the dataset we have are either daily or monthly. If the dataset contains monthly data, all timesteps can be inside one file. If the data are at a higher frequency, one file per month should get used, with the filename ending on *_YYYYMM. The parameter 'GEM_anclima' needs to get set to the location of the dataset to get used. A selection of already available datasets can get found under: ${MODEL_DATA}/SST_SeaIce_degK. But one can also create ones own dataset.

When only one file exists, 'GEM_anclima' needs to get set to the full name of the file.

If several monthly files exist, 'GEM_anclima' needs to get set to the full name of the file without the '_YYYYMM' at the end!!! The scripts will fill in the current year and month.

Lateral boundary conditions (LAM only)

When GEM is run in LAM mode, a set of atmospheric data need to get provided to drive the model at the lateral boundaries. These datasets need to include the following fields:

• Temperature
• Moisture (relative humidity, specific humidity, or dew point temperature)
• Horizontal winds in u- and v-grid(!) direction
• Geopotential (pressure levels: on all levels; model levels: only lowest level)
• Surface pressure
• Condensates (if available; they need to match the condensation scheme used!)

The 3-D fields can be either on model or on pressure levels.

Geophysical fields

Geophysical fields are time invariant fields describing surface properties like surface fraction (ocean, glacier, lakes, vegetation fractions), mountain height and several other fields.

To get a list of fields that need to get provided have a look at the following wiki: Geophysical fields

Schemes

All schemes get set in the 'gem_settings.nml'.

Land surface scheme (ISBA, CLASS, SVS)

One needs to choose a land surface scheme that calculates the heat fluxes and diagnostic fields over the land fraction. The land fraction includes bare soil (desert) and vegetated areas. If no urban model (see below) is used it also includes the urban fraction. The parameter 'schmsol' from the namelist 'surface_cfgs' needs to get set to one of the following: 
   'NIL '     : No Land surface processes (default)
   'ISBA'    : Interaction Soil Biosphere Atmosphere (ISBA) land sfc scheme
   'SVS '    : Soil, Vegetation, and Snow (SVS) (Multibudget) land sfc scheme
   'CLASS': Canadian LAnd Surface Scheme

ISBA

If ISBA is chosen, the following other parameters from the namelist 'surface_cfgs' also need to get set:

  !# If .true. apply temporary fix to ISBA
  !# * timestep dependent KCOEF
  !# * No PSN factor for meting and freezing
  logical           :: isba_melting_fix = .false.

  !# Use the vegetation-only roughness length to compute vegetation snow fraction
  logical           :: isba_snow_z0veg = .false. 

  !# If .true., freeze precipitation reaching the ground in sub-zero conditions
  logical           :: isba_zr_freeze = .false.

  !# Factor multiplying stomatal resistance in ISBA
  real              :: veg_rs_mult = 1.

  !# If .true. apply temporary fix to ISBA
  !# * timestep dependent KCOEF
  !# * No PSN factor for meting and freezing
  logical           :: isba_melting_fix = .false.

  !# Use the vegetation-only roughness length to compute vegetation snow fraction
  logical           :: isba_snow_z0veg = .false.

  !# Emissivity for bare soil (ISBA scheme only)
  !# * '_constant_' : A fixed floating point value used as a constant
  !# * 'CLIMATO'    : Value read from an input climatology file (EMIB)
  character(len=16) :: isba_soil_emiss = '0.95'
  real              :: isba_soil_emiss_const = -1.
  character(len=*), parameter :: ISBA_SOIL_EMISS_OPT(1) = (/ &
       'CLIMATO' &
       /)

  !# If .true., freeze precipitation reaching the ground in sub-zero conditions
  logical           :: isba_zr_freeze = .false. 

CLASS

If CLASS is chosen, the following other parameters from the namelist 'surface_cfgs' also need to get set:

  !# Number of soil layers in CLASS
  integer           :: class_ig    = 3

  !# Soil layer thickness in CLASS
  real              :: schmsol_lev(200) = 0.0

Lake scheme (non, FLake, CSLM)

One needs to choose a lake scheme that calculates the heat fluxes and diagnostic fields over the land fraction. The parameter 'schmlake' from the namelist 'surface_cfgs' needs to get set to one of the following: 

  'NIL'        : Lakes get treated my water routine (default)  =>  Prescribed SST and SIC will get used
  'FLAKE' : FLake lake scheme
  'CSLM'   : Canadian Small Lake Model

Urbain scheme (non, TEB)

One needs to choose a lake scheme that calculates the heat fluxes and diagnostic fields over the urban fraction. The parameter 'schmurb' from the namelist 'surface_cfgs' needs to get set to one of the following: 

  'NIL'  : No Urban surface processes (default)  =>  the urban fraction will get taken care of by the land surface model - see above
  'TEB' : Town Energy Balance (TEB) urban scheme

If 'TEB' is choosen, the following other parameters from the namelist 'surface_cfgs' also need to get set:

  !# Adjust wind diagnostic in TEB with building height  if .true.
  logical           :: urb_diagwind = .false.

  !# Adjust temperature diagnostic in TEB in the street  if .true.
  logical           :: urb_diagtemp = .false.

Emissivity

For the following surface fractions the emissivity can get adjusted in the namelist 'surface_cfgs':

    snow_emiss       : Emissivity for snow, default 1, recommended '0.97'
    water_emiss      : Emissivity for water, default 1, recommended '1.0'
    ice_emiss            : Emissivity for ice (glacier and sea ice), default '0.99', recommended '0.97'
    isba_soil_emiss : Emissivity for bare soil (ISBA scheme only), default '0.95', recommended 'climato'

Limit ice /snow

Due to biases, sea ice not drifting and glaciers not sliding downhill, sea ice and snow tend to get too think in certain places. Therefore, their depth can get limited by setting the parameters from the namelist 'surface_cfgs':

     ICEMAX           = 2.5 (recommended value)
     SNOWMAX    = 10. (recommended value)

Radiation

One needs to choose a radiation scheme. In general one chooses the latest one. Therefore, the parameter 'radia' from the namelist 'physics_cfgs' needs to get set to one of the following:

   'NIL'                   : no radiation scheme (default)
   'NEWRAD'       : complete radiation scheme
   'CCCMARAD'  : most advanced radiation scheme
   'CCCMARAD2': most advanced radiation scheme v2

When 'cccmarad2' is chosen, one should also set the following parameters from the namelist 'physics_cfgs' to .true.:

  !# Use climatological values of GHG in radiation (CCCMARAD2 only)
  logical           :: radghg_L     = .false.

  !# Use LINOZ prognostic Ozone in radiation (CCCMARAD2 .and. LINOZ only)
  logical           :: rad_linoz_L  = .false.

Turbulent orographic form drag

By default, part of the unresolved topography gets "added" as an orographic roughness length to the roughness length of the surface schemes. But instead one can also use a turbulent orographic form drag scheme. To do this the following parameters of the namelist 'physics_cfgs' can get set to:

    Tofd               = 'beljaars04'
    Tofd_alpha  = 35.

Gravity wave drag

The parameter 'gwdrag' from the namelist 'physics_cfgs' should get set to one of the following:

    'NIL'         : no Gravity wave drag (default)
    'GWD86' : gravity wave drag + low-level blocking
    'SGO16'  : new formulation (2016) of GWD86

The non-orographic GWD scheme, Hines, can get activated by setting the following parameter from the namelist 'physics_cfgs' to:

    non_oro  = .true.   (default .false.)

Condensation

To select a condensation scheme the parameter 'stcond' from the namelist 'physics_cfgs' should get set to one of the following:

   'NIL'                      : No explicit condensation scheme used (default)
   'CONSUN'           : Sunqvist type condensation scheme
   'NEWSUND'        : Sunqvist type condensation scheme
   'MP_MY2_OLD' : Milbrandtl and Yau microphysics scheme (old formulation)
   'MP_MY2'            : Milbrandtl and Yau microphysics scheme
   'MP_P3'               : P3 microphysics scheme

For resolutions of 10 km and lower we usually use 'consun'. For resolutions of 4 km and higher 'MP_P3'. However, 'MP_P3' can also get used at lower resolutions  (down to 12 km), especially when the output is used to drive a higher resolution simulation that uses 'MP_P3'.

When using stcond = 'MP_P3' one should also set the following parameters from the namelist 'gem_cfgs', for example to:

 p3_ncat          = 1
 p3_depfact   = 1.0
 p3_subfact   = 1.0
 p3_debug      = .false.
 p3_scpf_on   = .true.
 p3_pfrac        = 1.
 p3_resfact    = 0.46
 p3_clip_qv    = .true.

Tracers to advect

Depending on the condensation scheme, several tracers can get chosen to get advected by setting the parameter 'Tr3d_list_S' from the namelist 'gem_cfgs'. For example:

for 'stcond = consun':

 Tr3d_list_S  =  'HU,wload=0,mono=2,mass=1,hzd=0,min=0.',
                              'QC,wload=1,mono=2,mass=1,hzd=0,min=0.'

for 'stcond = MP_P3':

 Tr3d_list_S  =  'HU      ,  wload=0,mono=2,mass=1,hzd=0,min=0.',
                              'MPQC,  wload=1,mono=2,mass=1,hzd=0,min=0.',
                              'MPNC,  wload=0,mono=2,mass=1,hzd=0,min=0.',

Precipitation (Bourgouin)

To improve the precipitation type(!) (not the quantity) an extra scheme can get used by setting the parameter 'pcptype' from the namelist 'gem_cfgs' can get set to one of the following:

   'NIL'                  : no call to bourge (default)
   'BOURGE'       : use Bourgouin algorithm (bourge1) to determine precip. types - can sometimes improve the type of precipitation when using 'stcond = MP_P3'
   'BOURGE3D' : 3-D version of the Bourgouin algorithm - should get used when using 'stcond = consun'.

Boundary layer

A boundary layer scheme should get chosen by setting 'fluvert' from the namelist 'gem_cfgs' to one of the following:

   'NIL'              : no vertical diffusion (default)
   'CLEF'          : non-cloudy boundary layer formulation
   'MOISTKE' : cloudy boundary layer formulation
   'SIMPLE '    : a very simple mixing scheme for neutral PBLs

Usually, we are choosing either 'CLEF' or 'MOISTKE'.

When using fluvert = 'MOISTKE' one can also set the following parameter from the namelist 'gem_cfgs', for example to:

    Fnn_reduc  = 1.0           , 
    Fnn_mask   = .false.       ,

Mixing length

The mixing length calculation scheme should get set by setting the parameter 'longmel' from the namelist 'gem_cfgs' to one of the following:

    'BLAC62'         : mixing length calc. using Blackadar
    'BOUJO'          : mixing length calc. using Bougeault
    'TURBOUJO' : mixing length calc. using Bougeault in turbulent regimes (otherwise Blackadar)
    'LH'                   : mixing length calc. using Lenderink and Holtslag

Horizontal diffusion

Convection

The shallow convection scheme should get set by setting the parameter 'shal' from the namelist 'convection_cfgs' to one of the following:

    'NIL'                    (default)
    'KTRSNT'
    'KTRSNT_MG' (not recommended)
    'BECHTOLD'

The deep convection scheme should get set by setting the parameter 'deep' from the namelist 'convection_cfgs' to one of the following:

    'NIL' :
    'SEC' :
    'OLDKUO' :
    'KUOSTD' :
    'KFC' :
    'KFC2' :
    'KFC3' :
    'BECHTOLD' :

Output fields

Have a look at the following wiki page to learn more about the format of the file 'outcfg.out', in which all output fields need to get specified, apart from station data (see below): outcfg.out

One needs to define the output:

    grid             - choices are : model, core, free
    frequency - either in number of timesteps or hours
    level           - level type, either model or pressure levels

2-D output fields

Instantaneous / averages / min / max

If a field is an average (average), a minimum (min), a maximum (max), an accumulator (accum) or an accumulator to be averaged (avgacc) before output.

Initial condition fields

The output of fields that can later get used to initialize another simulation can get triggered by setting the parameter 'CLIMAT_out_anal' in the file configexp.cfg.

Pilot fields (2-D / 3-D)

The output of fields that can later get used to drive another simulation at the lateral boundaries can get triggered by setting the parameter 'CLIMAT_out_pilot' in the file configexp.cfg.

Size of output files (monthly/daily/hourly/...)

The following parameters of the namelist 'out' should get set to determine the size of the output files and the frequency at which the post processing should get triggered:

    'Out3_close_interval_S' - output file size: monthly, daily, hourly, minutely, secondly files or one file per timestep
    'Out3_postproc_fact'      - interval after how many output files the post processing should get triggered
    'Out3_pilot_unit_S'          - file size for case pilot files (nm_*): monthly, daily, hourly, minutely, secondly files or one file per timestep

Timeseries for certain points

The output of certain fields for specific points can get set by setting the following parameters in the namelist 'series':

    !# List of time series for surface variables
  character(len=SER_STRLEN_VAR) :: P_serg_srsrf_s(NVARMAX) = ' '

  !# List of time series for profile variables
  character(len=SER_STRLEN_VAR) :: P_serg_srprf_s(NVARMAX) = ' '

  !# Number of timesteps between time-series writeout
  integer :: P_serg_srwri = 1

  !# Times series package stops at this timestep
  integer :: P_serg_serstp = huge(1)

  !# Stations chosen in lat,lon for time-series
  !# Format: "STN1_NAME",lat1,lon1, "STN2_NAME",lat2,lon2, ...
  type(SER_STN_LALO_T) :: xst_stn_latlon(NSTATMAX)

Submission

Account

On clusters of the Alliance one needs to set under which allocation account a simulation should run. The default allocation account can get set in the file '~/.Account'. But it can get overwritten by exporting(!!!) the parameter 'RAPid' in the file configexp.cfg.

Wall clock time

On the clusters of the Alliance are different queues for jobs requesting different amounts of walltime as well as memory. To find out which type of queues exist on the cluster you want to run on check out the following wiki of the Alliance: Job_scheduling_policies

In the config file 'configexp.cfg' set 'BACKEND_time_mod' to the amount of seconds you want to request for a single job(1) - not for the whole simulation which can consist of multiple jobs.

In general, the shorter the requested time the shorter the queued time. Therefore, you want to request as little time as possible for a job. However, runtimes on clusters of the Alliance can vary by a lot, normally between -15 % to + 20 % of the average runtime. Hence, one should request runtimes that are about 25 % larger than the average expected runtime.

If you only want to run one job you can just request "enough" runtime. But when you are running a simulation with a sequence of jobs you might want to try to fit the jobs at the higher end of a queue. For Assuming, there are the following queues on a cluster:

• 3 hours or less,
• 12 hours or less,
• 24 hours (1 day) or less,
• 72 hours (3 days) or less,

And the simulation you want to run usually needs 14 hours per job.

Memory

By default, the maximum memory of the smallest compute nodes will get used. To use the default, the parameter 'BACKEND_cm' from the file configexp.cfg must be set to:
    BACKEND_cm=
But it can get overwritten with any other value. Accepted units are 'M' and 'G'.