The recipe below only works for simulations that were originally started with 'Chunk_lance'. If you need to restart a simulation originally started with 'Um_lance' contact me: winger.katja@uqam.ca
| Sommaire |
|---|
Restart a simulation
...
...
... from the beginning
To restart a simulation that already started running from the beginning you need to add the key '-start':
Chunk_lance -start
If you are starting a simulation for the very first time it is not necessary to add this key.
However, if your simulation already started but stopped in the executable of the first month and you want to make changes to the config files you need to restart the simulation from the beginning, so you need to add the key '-start'.
... from where it stopped
If the simulation was stopped (job was killed) or crashed it can often get re-submitted by executing again the command:
Chunk_lance
...
... from previous restart
...
files
Sometimes it happens that part of the restart files gets overwritten before a month is finished or that the restart files are corrupted or one would simply like to rerun of part of a simulation. In that case these cases one cannot just relaunch the month with Chunk_lance but has to get restart the original simulation from previous (uncorrupted) restart files back (from the month previous to the one one wants to rerun) and restart the simulation from there.When the model job stops for whatever reason, there is an automatic check in Chunk_lance if the restart files are still the original ones or not. If they got already modified the following message will appear in the "chunk_job listing" (!!!) not in the model listing:
At least one of the restart files got already rewritten
Therefore the model could not get restarted automatically
You have to restart your simulation starting from the previous restart files
----- ABORT -----
If you see this message one cannot simply restart the simulation with Chunk_lance but as said, one need to restart it from the previous restart file - see below.
To restart a simulation one need to :
.
Note: In the description below replace all text in cursive/italic and all '...' with the actual names!
...
Make sure the restart files of the previous month
...
are in the execution directory :
~/MODEL_EXEC_RUN/
...
${TRUE_HOST
...
}
...
3) Edit the file 'chunk_job.log'
To know where the simulation was, Chunk_lance is using a log file called 'chunk_job.log'. The chunk_job itself will check this file to determine which job to execute next. Therefore this log file should not be touched.
However, to rerun part of a simulation one can alter the log file by hand. Just make sure there is never a blank line at the end of the log file 'chunk_job.log' since the chunk_job only checks the very last line of the log file!
Restart using Chunk_lance
In case a simulation stops and you want to find out which job (scripts, entry or model) crashed, you have have look in the listings directory (~/listings/${TRUE_HOST}). Check which of the following jobs has crashed:
...
If the are not there you need to put them back. You will find all previous restart files under ${CLIMAT_archdir}/Restarts. To "put them back" you need to:
In any case, you can restart your simulation by simply executing
Chunk_lance
again in the config file directory.
Of course AFTER you fixed the problem - unless it was a machine problem. In the latter case, just restart the simulation with 'Chunk_lance'.
Restart from previous restart file
Continue simulation from a restart file:
- Copy restart file of previous month (if there is more than 1 part, copy all parts!) from the archive (
- Go into the execution directory:
cd ~/MODEL_EXEC_RUN/$TRUE_HOST - Untar the restart file from which you want to restart your simulation. With something like:
tar xvf
- ${CLIMAT_archdir}/Restarts
The cmcarc-command will create a new directory, but the *.ca file will remain in the directory. You can remove it again- /tar_file_for_previous_month.tarz
Click on the following link to learn more about restart files.
- Go into the execution directory:
- Go into the config file directory
- Edit the log file 'chunk_job.log':
(First I suggest to make a backup copy of the log file. Just in case.)
The Then remove all lines concerning the month you want to rerun and all following lines. - So the last line should contain something like:
... previous_month_M finished at ... If you are running the entry in parallel, you will also have to remove all the ${exp}_entry_finished flags in your config file directory for all the months you want to rerun. Otherwise the entries for these months will not get rerun!
Make sure the last line is not an empty line!
Click on the following link to learn more about the file 'chunk_job.log'.- In your config file directory you also need the script called:
${GEM_exp%_*}_month-to-rerun_S
If you do not have this script anymore in your config file directory you can find it in the archive in the file:
${CLIMAT_archdir}/Listings/jobs_....zip - Execute "Chunk_lance" again (without the '-start'!!!)