Restart Post Processing

When you are missing model output in your archive there is most of the time no need to rerun the simulation but only the post processing.

1) Check if the original model output is still where the model wrote it.

If the output is still there you will find it under the directory specified in ${CLIMAT_outrep}:

       ${CLIMAT_outrep}/${GEM_exp}/cfg_0000/*step_*/???-???/*

If the data are neither in your archive nor in the above directories you will have to rerun the month otherwise follow the steps below:

2) Make sure the "exper_configs" directory still exists with the file '.config_file.dot.cfg':

        ~/exper_configs/${GEM_exp}.generic/.config_file.dot.cfg

If the above file still exists go to step 3) below, otherwise put the file back. You should find the file in your zip file:

      ${CLIMAT_archdir}/Listings/configs_${GEM_exp%_*}.zip

You can unzip the whole ${GEM_exp}.generic directory and put it back under ~/exper_configs.

3) Check if the post processing job-script still exists:

      ~/delayed_jobs/${TRUE_HOST}/${GEM_exp}_DX_*

If you start the post processing more than once per month make sure you have the right job. Just look inside the jobs for the one with the correct "-firstdate ..." resp.  "-lastdate ...".
If the above file still exists go to step 4) below, otherwise put the file back. You should find the file in your zip file:

      ${CLIMAT_archdir}/Listings/jobs_${GEM_exp%_*}.zip

If you have more than one DX-job-script for the same month you need to unarchive the right one. The easiest way is to unarchive all of them and then look inside the jobs for the one with the correct "-firstdate ..." resp.  "-lastdate ...".

4) Once you have the data under ${CLIMAT_outrep}, the file '.config_file.dot.cfg' and the DX-script you can (re-)submit it.

You can find the command to submit the DX-job in your *_PP0_*  resp. *_PP_* listing. The submit command of the last DX-job is in the listing *_PP_* and if there were more DX-jobs the other submit commands are in the *_PP0_* listings. All the commands look like:

      soumet  DX-job   -t walltime   -listing listing_directory   -jn DX-job   -cm memory

All the DX-submit commands look the same. Once you found one you can submit all jobs with the same command. The only parameters changing are the two 'DX-job'.
If you want to submit a lot of DX-jobs you might want to submit them to the "super queue" by exporting the following variable once before submitting the DX-jobs:

      export QUEUE_1CPU=sj1@

Afterwards submit a super-job.