Some friendly advice to make your programming and your scripts/programs more efficient ...
- Keep all scripts/programs under your home to get them backed up automatically!
- Master the editor you are using. Know how to search (forwards/backwards, case sensitive/insensitive, whole words, ...)
- Write comments (so others have it easier to understand your code as well as your)
- Use indentation (mandatory in Python, also helps to read the code and find errors)
- Use colors (helps to read the code and find errors)
- Give names which have a “meaning” (for scripts/programs/variables). This also applies for email “subjects”!
- Verify your script/program section by section
- Add prints to make sure your script/program is really doing what you want it to do
- Chop your task up into smaller tasks - if possible. For example, if you want to plot daily means from hourly data, write one script to create the daily data. For this you can use r.diag/editfst (for RPN files) and CDO/NCO (for NetCDF files). Verify that the new files are correct. Then write a Matlab/Python script that only plots the data.
- Reduce memory usage:
- Matlab/Python: use 32-bit instead of 64-bit real/float numbers
- Do not load all data at the same time but only one file or even just one record at the time, treat it and then read the next record/file
- Reuse variables (this is more important than giving them meaningful names!) and/or
- delete large arrays once you finished using them
- monitor the cpu & memory usage of your jobs with 'top' - see below. - Make use of the cache:
Accessing data in the cache is up to 100 times faster than accessing data in memory
=> Once you read data, use them before reading more data
Once you created a variable which you have to use again do this as soon as possible (before it gets moved out of the cache) - When using nested loops make sure to use them in the right order.
In languages which do not need to get compiled, like Matlab and Python, loops over elements of large arrays should be avoided as they are very time consuming.
If you have to loop over all the elements of an array make sure you nest the loops in the right order. Let's say you want to loop over all the elements of the array humi(ni,nj,nk). Then the order of your loops should be:
- Close files once you are done reading/writing them
To check cpu and memory usage of active jobs on a server use the command 'top':
top
To only see your jobs add:
top -u ${USER}
By default, 'top' sorts all jobs by cpu usage. To sort them by memory usage just type 'M' (capital 'm') once 'top' is open.
To quit 'top' just press 'q'.