Set up the GEM model environment

You have to do the following only once.
If you set up the model environment on Compute Canada clusters, make sure you have already  ***set up the SSM environment***. If you have not done this already, click here for instructions. On the UQAM servers this has already been done during the creation of your account.

Basic setup (only needs to be done on clusters of The Alliance)

1) Open your directories for your group

If one day you like me to help you with anything on clusters of The Alliance, it would be really helpful if you would give read and execute access to your group for your home, project spaces and scratch space.
For the project spaces you can simply use ‘chmod’, since only members of the group can access the group directories. In the following commands replace ‘professor’ by the name of your PI (Principle Investigator):
    chmod g+rx ~/projects/def-professor/${USER}
    chmod g+rx ~/projects/rrg-professor/${USER}
    chmod g+rx ~/projects/ctb-professor/${USER}    # if existing

For your home and scratch space (for which the ‘group’ is yours and not the project group) you can do this using ACLs (Access Control Lists). For example:
    setfacl -m g:def-professor:r-x ~/.
    setfacl -m g:def-professor:r-x /scratch/${USER}
This will give access to all members of your group but not to anybody else.

2) Allow 'ssh $TRUE_HOST' without typing password:

Go into your directory ~/.ssh.

    cd ~/.ssh

If the directory ~/.ssh does not exist execute the command:
    ssh localhost
type in your password and once you are connected 'exit' again. You should now have the directory ~/.ssh. Change into it with : cd ~/.ssh

If there is no(!) file 'id_rsa_pub' in your ~/.ssh create it with:
    ssh-keygen         (press just 'Enter' whenever asked a question, 3 times)

Still in the directory ~/.ssh, add the public key to your authorized_keys with:
    cat id_rsa.pub >> authorized_keys
Afterwards make sure the permissions are correct with:
    chmod 644 ~/.ssh/authorized_keys

3) Add keys to your ~/.ssh/config to not get kicked out from a session

cat >> ~/.ssh/config << EOF
ForwardX11 no
stricthostkeychecking=no
ServerAliveInterval=15
ServerAliveCountMax=3
TCPKeepAlive=yes
UserKnownHostsFile=/dev/null
EOF

chmod 644 ~/.ssh/config

4) Create "host" name

Create a "host" with the name of the Compute Canada machine you are working on, pointing to 'localhost'
For example for Beluga create the following entry in your ~/.ssh/config (put lower case names):
cat >> ~/.ssh/config << EOF
#
Host beluga
Hostname localhost
EOF

On Narval put:
cat >> ~/.ssh/config << EOF
#
Host narval
Hostname localhost
EOF

5) Set ‘umask 022’

On newer Compute Canada clusters, like Cedar, Beluga and Narval, by default, everybody within the same group can modify and remove your data under the project space! To prevent this from happening set:
    umask 022
If you already set up the SSM environment, this command will be in your ~/.profile.d/.group_profile. But since the .group_profile is only executed for interactive processes and not for batch processes you should also add it to your batch profile:
    ~/.profile.d/.batch_profile
Create the file if you do not have it.

6) Set the core allocation name you want to use

When submitting any job on Compute Canada clusters you always need to specify which allocation account should get "billed" for the resources (cores) your job is running on.
Every PI (Principle Investigator - in general your professor) has a certain amount of cores assigned to her/his group which can get used on average(!) throughout the year. They are called "core years". Every PI has a default allocation (def-...) of 50 core years and some PIs also have a research project allocation (rrg-...) or even a contributed allocation (ctb-...), which vary in size from one PI to the next, from one cluster to the next and from one year to the next. To get an rrg-allocation the PI has to fill out an application once a year, describing the projects for which CC resources will get used. To get a ctb-allocation the PI has to buy the hardware.
You can find all the allocation names under which you can run under your directory:

    ~/projects

Pick the one you want to run your simulations under and write its name into a file (which you have to create) called:

    ~/.Account

This file has to contain only the account name under which you want to run, nothing else. You can find the accounts names of the different professors in the table below. In general I suggest you put the def-account of your professors as the default account in this file and target the RRG or CTB accounts for certain simulations directly.

For example:
    $ cat ~/.Account
    def-laprise

Set up the "GEM model environment"

Create model working directories

GEM needs a place to run and another to put its listings (log files) while the model is running. Both these places are hardcoded in the model to places under your home. But since your home only has a limited amount of space and the model needs lots of it, these two places need to be links to a place where you have more space and not actual directories.

First pick a place under which you want to put these model working directories. Set the variables 'model_space_L' and 'model_space_M' to the full path of the directories you pick to be able to use it for the rest of the setup:

• At UQAM: On the server on which you want to run GEM, create a directory under your assigned data space, for example:

      model_space_L=/file_system/${USER}/GEM
      model_space_M=/file_system/${USER}/GEM
  

  You need to replace 'file_system' by the name of the file system on which you have your data space.
  
  

• On Compute Canada systems:
  Because of the small quota for the number of files under the RRG project spaces and because these files are rather small, these two places should be links to your scratch space. However, since /scratch is often not working on Beluga and Narval I suggest you use your PI's project space instead., for example:

      model_space_L=/project/def-professor/${USER}/GEM
      model_space_M=/project/rrg-professor/${USER}/GEM    # If you PI does not have and rrg- or ctb-allocation use the default one, 'def-professor'

  You need to replace "professor" by the name of your professor. To find the right name have a look at the table above.
  

Then create the following directories/links:
(Whenever possible use copy -> paste to avoid typos and make sure the variables 'model_space_L' and 'model_space_M' are set - see above!!!)

    mkdir -p ${model_space_L}/Listings ~/listings
    ln -s ${model_space_L}/Listings ~/listings/${TRUE_HOST}

    mkdir -p ${model_space_M}/EXECDIR ~/MODEL_EXEC_RUN
    ln -s ${model_space_M}/EXECDIR ~/MODEL_EXEC_RUN/${TRUE_HOST}

After this you should have:

    ~/listings/${TRUE_HOST} -> ${model_space_L}/Listings
    ~/MODEL_EXEC_RUN/${TRUE_HOST} -> ${model_space_M}/EXECDIR

Create “Storage_model”

When creating your executables the object files, extracted decks and executables will be put in a directory called ${storage_model}. A link will automatically be added to this place from the directory in which you create the executables.

You have to create this directory in a place in which you have space (therefore not under your home) and export the variable "storage_model" set to that directory.

• At UQAM:
  Set this directory to the same directory under which you created your working directories above:

      export storage_model=/file_system/${USER}/GEM/Storage_Model
  
  

• On Compute Canada systems:
  Because of the small quota for the number of files under the RRG project spaces and because these files are rather small, set this directory to a place under the default project (def-professor) space:
  

      export storage_model=~/projects/def-professor/${USER}/GEM/Storage_Model

  You need to replace "def-professor" by the name of the default account of your professor. To find the right name have a look at the table above.
  Note: Whereas the model working directories can be under your scratch space, your "storage" space must be in a permanent place!

Add this export to these two profiles (if you do not have them, create them):

    ~/.profile.d/.interactive_profile
    ~/.profile.d/.batch_profile

and create the following symbolic link:

    ln -s ~/.profile.d/.batch_profile ~/.profile.d/.ssh_profile

Log out and back in.

Then create the directory with:

    mkdir -p ${storage_model}