...
If the model stopped in the Fortran part, most of the time you can find an error message at the end of the listing of process 0. To get there jump to the end of the listing and then search backwards for the end of the listing of the main process. (When using 'vi', 'vim' or 'less' you can jump to the end by pressing 'G' and then search upward with '?00000:'). But even from the end of the main model listing you might still have to look several lines up to find an error. However, once you reached a line saying:
THE TIME STEP n IS COMPLETED
there is probably no error above anymore and then you will have to look into the listings of all the other processes.
When the Fortran part finishes fine, you will see the following messages at the end of the main process listing:
oe-00000-00000: Memory==> ...
:
oe-00000-00000: __________________TIMINGS ON PE #0_________________________
:
oe-00000-00000: .........RESTART
And then a big '****' box with an "END EXECUTION" inside.
INFO: END of listing processing : Mon Aug 30 17:13:58 EDT 2021
INFO: RUN FAILED
INFO: first 10 failing processes :
fail.00000-00062.00062 fail.00000-00103.00103
3) Second shell part
============== end of parallel run ==============
INFO: END of listing processing : Mon Aug 30 12:33:56 EDT 2021
INFO: END of listing processing : Mon Aug 30 17:13:58 EDT 2021
INFO: RUN FAILED
INFO: first 10 failing processes :
fail.00000-00062.00062 fail.00000-00103.00103