General information

Domain & resolution

First you should decide upon the final(!) (see "Cascading" below) domain and resolution you want to run. For this it is important to know that:

• the more grid points a grid has => the more cores are needed
• the higher the resolution => the shorter the model timestep => the longer the runtime of the simulation

Have a look at the following wiki about how to set up a model grid: Set up a model grid

Topology (MPI & OpenMP)

Resolution → timestep

The relation between the horizontal resolution and the model timestep is linear. Here are some examples:

Resolution → timestep → radiation timestep

Calculating the increments on different fields depending on the radiation is rather costly. Therefore, the option exists to calculate the increments only every n seconds but still apply them at every timestep. The following parameters of the namelist 'physics_cfgs' can get set to a multiple 4 to 6 of the model timestep.

     'kntrad_S'      : Time between full radiation calculation (units D,H,M,S,P)
     'kntraduv_S' : Time between UV radiation calculation (units D,H,M,S,P)

Resolution → (non) hydrostatic

If the grid resolution is ~10 km or lower the model is hydrostatic and one should set 'Dynamics_hydro_L = .true.' in the namelist 'dyn_kernel'. For grid resolutions of ~4km and higher one should set 'Dynamics_hydro_L = .false.'

Model levels

The model levels on GEM are terrain following at the surface and pressure levels at the model top. The model levels are generally closer to each other at the surface and further apart higher up. The model is very sensitive to the location of these model levels. They must neither be too far apart nor too close and the transition from terrain following to pressure levels must be very smooth. There are ways to set up these model levels. Several sets of model levels have already been created and can get selected by setting 'CLIMAT_etaname' in the file configexp.cfg. The possible settings for this parameter can get found in the file:
      ~/modeles/GEMDM/*/bin/Climat_eta2preshyb

Vertical sponge & top piloting (LAM only)

Probably because waves get reflected back into the atmosphere from the model lid, a spurius wind jet forms at the equatorial stratopause. To remove this spurius wind jet, a so called "vertical sponge" is used at the model top to absorb these waves so they do not get reflected. For more information and how to set up the vertical sponge (Vspng) have a look at the following wiki: Diffusion & Sponges

In LAM model one also has the option of instead nesting the top model layers. Which is also called "top piloting".

When top piloting is used the vertical sponge should NOT get used!!!

Global grid → adjust global pressure

Due to the semi-lagrangian advection scheme, which does not exactly conserve the total mass of the atmosphere, the GEM model is "loosing air" with time. Therefore, the parameter 'Schm_psadj' from the namelist 'dyn_fisl' needs to get set to one of the following:

      !# * 0 -> No conservation of surface pressure (for LAM grids)
      !# * 1 -> Conservation of total air mass pressure (for global grids)
      !# * 2 -> Conservation of dry    air mass pressure (for global grids)

Cascadeing

When the resolution of the driving data is too large with respect to the model grid resolution, one or more simulations can get run with an intermediate resolution and a domain slightly larger than the next inner domain.

Spectral nudging

To constrain the large scales of a simulation spectral nudging can get used. To learn more about this and which parameters to set have a look at the following wiki page: Spectral nudging

Period

Start and end date of a simulation need to get set in the file configexp.cfg with the parameters:
    CLIMAT_startdate="YYYY MM DD hh"
    CLIMAT_enddate="YYYY MM DD hh"

Greenhouse gases / Scenario

Depending on the period (past, current, future) and for future simulation depending on the scenario simulated, a table containing the annual mean concentrations of the main 5 greenhouse gases (CO2, N2O, CH4, CFC-11, CFC-12) needs to get assigned to the parameter 'CLIMAT_ghg_list' (in file configexp.cfg). A selection of different lists can be found under: ${MODEL_DATA}/Greenhouse_gases

But one can also create their own list.

Initial conditions

At the beginning of every simulations several fields (atmospheric, surface and soil) need to get initialized. Especially for short simulations it is very important to use fields that best represent the starting state of a simulation. If such fields are not available one can also choose to spinup the simulation. Whereas atmospheric fields get into equilibrium very quickly, surface fields like snow on the ground might take up to at least a season. Soil fields, like soil temperature and soil moisture, can take several years to get into equilibrium, depending on the soil depth.

To get a list of fields that need to get initialized have a look at the following wiki: Initial conditions

Driving data (lower & lateral)

Lower boundary conditions (SST & SIC)

Since our(!!!) GEM version, CRCM-GEM, is not coupled to an ocean model, the sea surface temperature (SST) as well as the sea ice fraction (SIC) need to get prescribed. These two fields can get prescribed at every model timestep up to monthly fields. Most of the dataset we have are either daily or monthly. If the dataset contains monthly data, all timesteps can be inside one file. If the data are at a higher frequency, one file per month should get used, with the filename ending on *_YYYYMM. The parameter 'GEM_anclima' needs to get set to the location of the dataset to get used. A selection of already available datasets can get found under: ${MODEL_DATA}/SST_SeaIce_degK. But one can also create ones own dataset.

When only one file exists, 'GEM_anclima' needs to get set to the full name of the file.

If several monthly files exist, 'GEM_anclima' needs to get set to the full name of the file without the '_YYYYMM' at the end!!! The scripts will fill in the current year and month.

Lateral boundary conditions (LAM only)

When GEM is run in LAM mode, a set of atmospheric data need to get provided to drive the model at the lateral boundaries. These datasets need to include the following fields:

• Temperature
• Moisture (relative humidity, specific humidity, or dew point temperature)
• Horizontal winds in u- and v-grid(!) direction
• Geopotential (pressure levels: on all levels; model levels: only lowest level)
• Surface pressure
• Condensates (if available; they need to match the condensation scheme used!)

The 3-D fields can be either on model or on pressure levels.

Geophysical fields

Geophysical fields are time invariant fields describing surface properties like surface fraction (ocean, glacier, lakes, vegetation fractions), mountain height and several other fields.

To get a list of fields that need to get provided have a look at the following wiki: Geophysical fields

Schemes

All schemes get set in the 'gem_settings.nml'.

Land surface scheme (ISBA, CLASS, SVS)

One needs to choose a land surface scheme that calculates the heat fluxes and diagnostic fields over the land fraction. The land fraction includes bare soil (desert) and vegetated areas. If no urban model (see below) is used it also includes the urban fraction. The parameter 'schmsol' from the namelist 'surface_cfgs' needs to get set to one of the following: 
   'NIL '     : No Land surface processes (default)
   'ISBA'    : Interaction Soil Biosphere Atmosphere (ISBA) land sfc scheme
   'SVS '    : Soil, Vegetation, and Snow (SVS) (Multibudget) land sfc scheme
   'CLASS': Canadian LAnd Surface Scheme

ISBA

If ISBA is chosen, the following other parameters from the namelist 'surface_cfgs' also need to get set:

  !# If .true. apply temporary fix to ISBA
  !# * timestep dependent KCOEF
  !# * No PSN factor for meting and freezing
  logical           :: isba_melting_fix = .false.

  !# Use the vegetation-only roughness length to compute vegetation snow fraction
  logical           :: isba_snow_z0veg = .false. 

  !# If .true., freeze precipitation reaching the ground in sub-zero conditions
  logical           :: isba_zr_freeze = .false.

  !# Factor multiplying stomatal resistance in ISBA
  real              :: veg_rs_mult = 1.

  !# If .true. apply temporary fix to ISBA
  !# * timestep dependent KCOEF
  !# * No PSN factor for meting and freezing
  logical           :: isba_melting_fix = .false.

  !# Use the vegetation-only roughness length to compute vegetation snow fraction
  logical           :: isba_snow_z0veg = .false.

  !# Emissivity for bare soil (ISBA scheme only)
  !# * '_constant_' : A fixed floating point value used as a constant
  !# * 'CLIMATO'    : Value read from an input climatology file (EMIB)
  character(len=16) :: isba_soil_emiss = '0.95'
  real              :: isba_soil_emiss_const = -1.
  character(len=*), parameter :: ISBA_SOIL_EMISS_OPT(1) = (/ &
       'CLIMATO' &
       /)

  !# If .true., freeze precipitation reaching the ground in sub-zero conditions
  logical           :: isba_zr_freeze = .false. 

CLASS

If CLASS is chosen, the following other parameters from the namelist 'surface_cfgs' also need to get set:

  !# Number of soil layers in CLASS
  integer           :: class_ig    = 3

  !# Soil layer thickness in CLASS
  real              :: schmsol_lev(200) = 0.0

Lake scheme (non, FLake, CSLM)

One needs to choose a lake scheme that calculates the heat fluxes and diagnostic fields over the land fraction. The parameter 'schmlake' from the namelist 'surface_cfgs' needs to get set to one of the following: 

  'NIL'        : Lakes get treated my water routine (default)  =>  Prescribed SST and SIC will get used
  'FLAKE' : FLake lake scheme
  'CSLM'   : Canadian Small Lake Model

Urbain scheme (non, TEB)

One needs to choose a lake scheme that calculates the heat fluxes and diagnostic fields over the urban fraction. The parameter 'schmurb' from the namelist 'surface_cfgs' needs to get set to one of the following: 

  'NIL'  : No Urban surface processes (default)  =>  the urban fraction will get taken care of by the land surface model - see above
  'TEB' : Town Energy Balance (TEB) urban scheme

If 'TEB' is choosen, the following other parameters from the namelist 'surface_cfgs' also need to get set:

  !# Adjust wind diagnostic in TEB with building height  if .true.
  logical           :: urb_diagwind = .false.

  !# Adjust temperature diagnostic in TEB in the street  if .true.
  logical           :: urb_diagtemp = .false.

Emissivity

For the following surface fractions the emissivity can get adjusted in the namelist 'surface_cfgs':

    snow_emiss       : Emissivity for snow, default 1, recommended '0.97'
    water_emiss      : Emissivity for water, default 1, recommended '1.0'
    ice_emiss            : Emissivity for ice (glacier and sea ice), default '0.99', recommended '0.97'
    isba_soil_emiss : Emissivity for bare soil (ISBA scheme only), default '0.95', recommended 'climato'

Limit ice /snow

Due to biases, sea ice not drifting and glaciers not sliding downhill, sea ice and snow tend to get too think in certain places. Therefore, their depth can get limited by setting the parameters from the namelist 'surface_cfgs':

     ICEMAX           = 2.5 (recommended value)
     SNOWMAX    = 10. (recommended value)

Radiation

One needs to choose a radiation scheme. In general one chooses the latest one. Therefore, the parameter 'radia' from the namelist 'physics_cfgs' needs to get set to one of the following:

   'NIL'                   : no radiation scheme (default)
   'NEWRAD'       : complete radiation scheme
   'CCCMARAD'  : most advanced radiation scheme
   'CCCMARAD2': most advanced radiation scheme v2

When 'cccmarad2' is chosen, one should also set the following parameters from the namelist 'physics_cfgs' to .true.:

  !# Use climatological values of GHG in radiation (CCCMARAD2 only)
  logical           :: radghg_L     = .false.

  !# Use LINOZ prognostic Ozone in radiation (CCCMARAD2 .and. LINOZ only)
  logical           :: rad_linoz_L  = .false.

Turbulent orographic form drag

By default, part of the unresolved topography gets "added" as an orographic roughness length to the roughness length of the surface schemes. But instead one can also use a turbulent orographic form drag scheme. To do this the following parameters of the namelist 'physics_cfgs' can get set to:

    Tofd               = 'beljaars04'
    Tofd_alpha  = 35.

Gravity wave drag

The parameter 'gwdrag' from the namelist 'physics_cfgs' should get set to one of the following:

    'NIL'         : no Gravity wave drag (default)
    'GWD86' : gravity wave drag + low-level blocking
    'SGO16'  : new formulation (2016) of GWD86

The non-orographic GWD scheme, Hines, can get activated by setting the following parameter from the namelist 'physics_cfgs' to:

    non_oro  = .true.   (default .false.)

Condensation

To select a condensation scheme the parameter 'stcond' from the namelist 'physics_cfgs' should get set to one of the following:

   'NIL'                      : No explicit condensation scheme used (default)
   'CONSUN'           : Sunqvist type condensation scheme
   'NEWSUND'        : Sunqvist type condensation scheme
   'MP_MY2_OLD' : Milbrandtl and Yau microphysics scheme (old formulation)
   'MP_MY2'            : Milbrandtl and Yau microphysics scheme
   'MP_P3'               : P3 microphysics scheme

For resolutions of 10 km and lower we usually use 'consun'. For resolutions of 4 km and higher 'MP_P3'. However, 'MP_P3' can also get used at lower resolutions  (down to 12 km), especially when the output is used to drive a higher resolution simulation that uses 'MP_P3'.

Tracers to advect

Depending on the condensation scheme, several tracers can get chosen to get advected by setting the parameter 'Tr3d_list_S' from the namelist 'gem_cfgs'. For example:

for 'stcond = consun':

 Tr3d_list_S  =  'HU,wload=0,mono=2,mass=1,hzd=0,min=0.',
                              'QC,wload=1,mono=2,mass=1,hzd=0,min=0.'

for 'stcond = MP_P3':

 Tr3d_list_S  =  'HU,  wload=0,mono=2,mass=1,hzd=0,min=0.',
                              'MPQC,  wload=1,mono=2,mass=1,hzd=0,min=0.',
                              'MPNC,  wload=0,mono=2,mass=1,hzd=0,min=0.',

Precipitation (Bourgouin)

Boundary layer

Horizontal diffusion

Convection

Output fields

Have a look at the following wiki page to learn more about the format of the file 'outcfg.out', in which all output fields need to get specified, apart from station data (see below): outcfg.out

One needs to define the output:

    grid             - choices are : model, core, free
    frequency - either in number of timesteps or hours
    level           - level type, either model or pressure levels

2-D output fields

Instantaneous / averages / min / max

If a field is an average (average), a minimum (min), a maximum (max), an accumulator (accum) or an accumulator to be averaged (avgacc) before output.

Initial condition fields

The output of fields that can later get used to initialize another simulation can get triggered by setting the parameter 'CLIMAT_out_anal' in the file configexp.cfg.

Pilot fields (2-D / 3-D)

The output of fields that can later get used to drive another simulation at the lateral boundaries can get triggered by setting the parameter 'CLIMAT_out_pilot' in the file configexp.cfg.

Size of output files (monthly/daily/hourly/...)

The following parameters of the namelist 'out' should get set to determine the size of the output files and the frequency at which the post processing should get triggered:

    'Out3_close_interval_S' - output file size: monthly, daily, hourly, minutely, secondly files or one file per timestep
    'Out3_postproc_fact'      - interval after how many output files the post processing should get triggered
    'Out3_pilot_unit_S'          - file size for case pilot files (nm_*): monthly, daily, hourly, minutely, secondly files or one file per timestep

Timeseries for certain points

The output of certain fields for specific points can get set by setting the following parameters in the namelist 'series':

    !# List of time series for surface variables
  character(len=SER_STRLEN_VAR) :: P_serg_srsrf_s(NVARMAX) = ' '

  !# List of time series for profile variables
  character(len=SER_STRLEN_VAR) :: P_serg_srprf_s(NVARMAX) = ' '

  !# Number of timesteps between time-series writeout
  integer :: P_serg_srwri = 1

  !# Times series package stops at this timestep
  integer :: P_serg_serstp = huge(1)

  !# Stations chosen in lat,lon for time-series
  !# Format: "STN1_NAME",lat1,lon1, "STN2_NAME",lat2,lon2, ...
  type(SER_STN_LALO_T) :: xst_stn_latlon(NSTATMAX)

Submission

Account

On clusters of the Alliance one needs to set under which allocation account a simulation should run. The default allocation account can get set in the file '~/.Account'. But it can get overwritten by exporting(!!!) the parameter 'RAPid' in the file configexp.cfg.

Wall clock time

On the clusters of the Alliance are different queues for jobs requesting different amounts of walltime as well as memory. To find out which type of queues exist on the cluster you want to run on check out the following wiki of the Alliance: Job_scheduling_policies

In the config file 'configexp.cfg' set 'BACKEND_time_mod' to the amount of seconds you want to request for a single job(1) - not for the whole simulation which can consist of multiple jobs.

In general, the shorter the requested time the shorter the queued time. Therefore, you want to request as little time as possible for a job. However, runtimes on clusters of the Alliance can vary by a lot, normally between -15 % to + 20 % of the average runtime. Hence, one should request runtimes that are about 25 % larger than the average expected runtime.

If you only want to run one job you can just request "enough" runtime. But when you are running a simulation with a sequence of jobs you might want to try to fit the jobs at the higher end of a queue. For Assuming, there are the following queues on a cluster:

• 3 hours or less,
• 12 hours or less,
• 24 hours (1 day) or less,
• 72 hours (3 days) or less,

And the simulation you want to run usually needs 14 hours per job.

Memory

By default, the maximum memory of the smallest compute nodes will get used. To use the default, the parameter 'BACKEND_cm' from the file configexp.cfg must be set to:
    BACKEND_cm=
But it can get overwritten with any other value. Accepted units are 'M' and 'G'.