...
Have a look at the following wiki about how to set up a model grid: Set up a model grid
Topology (MPI & OpenMP)
To speed up a simulation one can run it on more then one cpu. Or in other words, one can "run it in parallel". There are two ways to parallelize GEM simulations: MPI and OpenMP. Both get used at the same time by setting the parameter 'GEM_ptopo' from the file configexp.dot:
GEM_ptopo=(MPI-x)x(MPI-y)x(OpenMP) # See more explanations below
If more than 1 node will get used, the total number of cpus used should be a multiple of the number of cores of a compute node!
For Yin-Yang grids, the number of cores specified in 'GEM_ptopo' is the number used for each of the LAM grids. So the total number will be twice as high!
MPI (Message Passing Interface)
Using MPI to parallelize GEM is more efficient than using OpenMP. To parallelize the a simulation using MPI the full model domain (including pilot region) gets divided into "tiles". Each MPI process will then take care of one tile. Meaning there will be as many MPI processes as there will be tiles.
To use MPI one needs to set the first two numbers of 'GEM_ptopo' to the number of tiles one would like to use in x- (MPI-x) and y-direction (MPI-y). For example, to use 2 tiles in x- and 2 tiles in y-direction set:
GEM_ptopo=2x2x1
FYI, if the total number of points of the model grid is not exactly dividable by the number of MPI tiles, the first tile(s) will get more points and the other(s) less.
Basic rules to consider
- The minimum tile size is 23x23 points!
- The maximum tile size depends on the schemes used: The "memory heavier" a scheme, the smaller the tiles need to be. For example, 'P3' needs more memory than 'consun' and when adding 'TEB' the simulation will also need much more memory. In general, a maximum tile size of about 100x100 points should not get exceeded.
- Tiles should be as square as possible to minimize the information exchange over the tile borders.
- If tiles cannot be square and one has a choice, it is preferable to have more tiles in y- than in x-direction to help the solver.
OpenMP
In GEM the physics calculate one "slab" (row * levels) after the other. When using OpenMP, more than one slab will get treated a the same time. To parallelize the a simulation by using OpenMP one needs to set the last number of 'GEM_ptopo' to the number of OpenMP processes to get used.
It is NOT recommended to use more than two OpenMP processes. To use OpenMP=2 in the above example one would set:
GEM_ptopo=2x2x2
Resolution → timestep
The relation between the horizontal resolution and the model timestep is linear. Here are some examples:
...