The recipe below only works for simulations that were originally started with 'Chunk_lance'. If you need to restart a simulation originally started with 'Um_lance' contact me: winger.katja@uqam.ca

Sommaire

Restart

...

a simulation ...

... from the beginning

To restart a simulation that already started running from the beginning you need to add the key '-start':

    Chunk_lance -start

If you are starting a simulation for the very first time it is not necessary to add this key.
However, if your simulation already started but stopped in the executable of the first month and you want to make changes to the config files you need to restart the simulation from the beginning, so you need to add the key '-start'.

... from where it stopped

If the simulation was stopped (job was killed) or crashed it can often get re-submitted by executing again the command:

    Chunk_lance

...

... from previous restart

...

files

Sometimes it happens that part of the restart files gets overwritten before a month is finished or that the restart files are corrupted or one would simply like to rerun part of of a simulation. In that case these cases one cannot just relaunch the month with Chunk_lance but has to get the original (uncorrupted) restart files back (from the month previous to the one one wants to rerun) and restart the simulation from there.When the model job stops for whatever reason, there is an automatic check in Chunk_lance if the restart files are still the original ones or not. If they got already modified the following message will appear in the "chunk_job listing" (!!!) not in the model listing:
   At least one of the restart files got already rewritten
   Therefore the model could not get restarted automatically
   You have to restart your simulation starting from the previous restart files
            ----- ABORT -----
 
If you see this message one cannot simply restart the simulation with Chunk_lance but as said, one need to restart it from the previous restart file - see below.

To restart a simulation one needs to :

previous (uncorrupted) restart files:

...

1. Make sure the restart files of the previous month

...

1. are

...

• gem_restart  
• gmm_restart  
• Whiteboard.ckpt

...

1. in

...

1. While the model is running you can find them under:

...

1. the execution directory :
          ~/MODEL_EXEC_RUN/${TRUE_HOST}

...

1. 
   If the are not there you need to put them back. For example with:

   Volet
   cd

...

1. ~/MODEL_EXEC_RUN/${TRUE_HOST}

...

1. tar xvf   /path/to/archive/Restarts/tar_file_for_previous_month.tar

   You need to adjust 'path/to/archive' as well as 'tar_file_for_previous_month'.
   Click on the following link to learn more about restart files.

2. Go into the config file directory
3. Edit the log file 'chunk_job.log':
   (First I suggest to make a backup copy of the log file.)
   Then remove all lines concerning the month you want to rerun and all following lines.
   Click on the following link to learn more about the file 'chunk_job.log'.
4. Make sure the "Sripts

2. Once the model simulation for a month has finished but the post processing did not run yet they are under:

    ~/MODEL_EXEC_RUN/${TRUE_HOST}/Restarts/${GEM_exp}/*step*/*/RUNMOD/work/cfg_0000/...

3. Once the post processing finished the "archiving of the restarts" they will be in your archive under:

    ${CLIMAT_archdir}/Restarts

Depending on the script version you are using they can have different names. Either

...

Gunzip and unarchive (cmcarc -x -f ...) the restart file(s) in ~/MODEL_EXEC_RUN/$TRUE_HOST
Depending on the grid size there could be more than one file per month. You need to gunzip and unarchive all of them.
The cmcarc-command will create a new directory, but the *.ca file(s) will remain in the directory. You can remove it/them again.

or

...

1. -job", ${GEM_exp}_S, is in the config directory
   In case

...

1. this file is not in your config directory anymore you can find it in your archive under:
          ${CLIMAT_archdir}/Listings/jobs_*.zip

...

3) Edit the file 'chunk_job.log'

As you hopefully know, each simulated month consists of two scripts:

• a very short                             Script-job : ${GEM_exp}_S
• followed by one of more Model-jobs : ${GEM_exp}_M

To know where the simulation is, Chunk_lance is using a log file called 'chunk_job.log'. The chunk_job itself will check this file to determine which job to execute next. Therefore, this log file should usually not be touched. However, to rerun part of a simulation one can alter the log file by hand. Just make sure there is never a blank line at the end of the log file 'chunk_job.log' since the chunk_job only checks the very last line of the log file!

'chunk_job.log' contains entries like the following:

...

...

Each completed month has at least 4 entries:

To restart a month make sure the last line either contains:

or

1. Execute "Chunk_lance" again (without the '-start'!!!)