Working on the servers of the ESCER Centre

Our computer infrastructure

Have a look at our computer infrastructure: Computer infrastructure of the ESCER Centre

Firewall

To protect our servers and data we have a firewall installed around all of them. To get inside this firewall you need to either:

• be connected via cable on the internet on the 6st or 7th floor of the PK
• use UQAM's VPN
• connect to one of our hoppies - see "External servers" below.

External servers

The so called "external" servers are outside our firewall and are only used to connect to our internal servers when connecting from outside the firewall. You can find their names here : external servers

Internal servers

We call them "internal" because they are inside our firewall. 

To access your/our data you need to be on one of our "internal servers". Even though each user has access to all of our servers, please, use the server "assigned" to your professor. The name of the server you should use is also written in the email you received about your account. If one day you switch professors, please, let me (winger.katja@uqam.ca) know!

Check out our full internal server pool under : Internal servers

Also, have a look at the graphic on the following site that visualizes all of the above : Computer infrastructure of the ESCER Centre

Operating system 

The operating system on our servers is Linux. If you are not familiar with Linux/UNIX have a look a the basic commands: Basic Linux/UNIX commands

Connection to internal servers

When using a workstation or laptop in an office on the 6st or 7th floor in the PK, which is connected to the internet via cable, you can connect to our internal servers directly. Otherwise you will have to connect via a VPN or via one of our external servers hoppy1/hoppy2/hoppy3.
To learn more about how to connect to our servers check out the following wiki page :  Connexion à nos serveurs

Change password

To change your password, follow the instructions on our wiki page:    Change password

Available disk space, memory and CPU power

Have a look at the following wiki page to find out where to put your data and how much memory and CPU power you can use:

    Disk space, memory and CPU power

Data transfer

If ever you need to transfer data from/to one of the internal servers to/from the outside, follow the instructions on the following page:   Transfert de données de/vers l'extérieur

The software

On our UQAM servers, as well as on clusters of The Alliance, 'modules' are used to give access to different software packages. Learn more about modules on the following page:   Modules

Python

Even though there are default Python version installed on some of the servers it is strongly recommended to load a version via the modules (see above):   Accéder à Python

Jupyter

If you want to use Jupyter follow the instruction on the following site:   Accéder à Jupyter

RPN Python

To be able to read/write RPN files in Python, follow the instruction on the following site:   RPN Python

Aliases & exports

If you want to create and add your own aliases and exports you need to put them in the following file:

    ~/.profile.d/.interactive_profile    # For interactive processes
    ~/.profile.d/.batch_profile              # For batch processes

Running jobs in batch

If you want to run jobs that run for a long time and you would like to be able to logout you can submit jobs to our queuing system, which will take care of running your job. Then you can log out. The job will continue running as usual. Have a look at how to run jobs in batch on our servers (and on clusters of the Alliance if using our environment): Batch processes

CRCM/GEM

At the ESCER Centre we a using a slightly modified version of ECCC's (Environnement Canada et Climat Canada) weather forecast model GEM (Global Environmental Multiscale), officially called "Developmental version of CRCM/GEM".
For more information click o the following link:   CRCM/GEM

GEM model output

Instead of being in NetCDF or GRIB format, the GEM model in- and output is the RPN standard format. This format comes with it's own set of tools to look at and to modify the data.

On the internal servers of the ESCER Centre you can find some practice data under:

    ~data/Example_data

You can find some general information about our GEM model output here:   GEM model output

Full description of the tools below and other tools can also be found on the following page:  RPN standard format

Having a first glance at the GEM model output

vl3 / vl4 / vl5 → list variable names and descriptions

To list the names (and if lucky units) of the variables inside a file try the command:

And there are some more variable lists here:    Variable dictionaries

voir → list content (names / date/time / level)

To have a closer look at the fields and see date/time, levels, etc. try 'voir':

For more information about 'voir' check here:   voir - documentation

xrec → visualize fields

To visualize the fields you can use 'xrec':

But you need to have connected with 'ssh -YC ...' to open windows.
For more information about 'xrec' check here as well:    xrec - documentation

Manipulate RPN files

To extract fields you can use either 'r.diag select ...' or 'editfst'. You can find information about both here:   Manipulating RPN files
But both these tools can be a little tricky to use. So don't hesitate to contact me when you get there!

Convert RPN to NetCDF

In general it would be better to keep GEM output in RPN format to avoid "doubling" the output data. However, in some cases it can be useful to convert files in RPN format to NetCDF format or vice versa.
The tool with which one can do this is called cdf2rpn.  You can find information about cdf2rpn under the following link:    cdf2rpn - documentation

Running GEM

CRCM/GEM is installed on certain clusters of The Digital Research Alliance of Canada. 

To learn how to run CRCM/GEM have a look at the wiki pages under: Running CRCM/GEM

Courses

SciNet, the High Performance Computing Consortium at the University of Toronto, is offering a large variety of SciNet online courses.