A rather annoying "feature" of running calculations on a remote host, such as a cluster, is that if you are not careful, the calculation will be interrupted if you lose the network connection.
The way I used to get around this was to submit my jobs under "nohup", like
nohup atkpython myscript.py > myscript.log &
Note the ampersand "&" at the end, so that the processes goes into the background.
Now, there are certain disadvantages with nohup, which I will not go into. One very notable feature is also that if you launch "mpiexec" under nohup, you must redirect stdin too:
nohup mpiexec atkpython myscript.py > myscript.log < /dev/null &
otherwise your log file will fill up with warning messages real fast.
Fortunately, there is an alternative to nohup, called screen
. There is a very nice tutorial for it here:http://www.rackaid.com/resources/linux-screen-tutorial-and-how-to/
To summarize briefly how to run ATK on a detached screen (we assume screen is installed on the remote host; if not, please confer with the sysadm of that system):
1. Log into the remote host where you want to run, e.g. using ssh
2. Use the command "screen" to start a new screen (there is some information screen that pops up, it can be avoided with "screen -q")
3. Start ATK as you would normally, for instance
(we use an example without a log file, it works fine with one as well, of course)
4. Hit Ctrl-A-D to detach the screen
5. You will be returned to the login session on the node, from which you can now safely log off ("exit") and the calculation will continue to run in the background.
Some time later, you want to check if the calculation finished.
1. ssh back into the remote host
2. Type "screen -r" to resume the screen (if you have several screens running, you will be prompted to provide the pid, like "screen -r 1234")
3. You are now back in the session where you started ATK, and you can operate as usual. If the calculation didn't finish, just detach the screen with Ctrl-A-D again. If it finished and you want to exit permanently, type "exit"
There are more options, like switching between different screens (Ctrl-A-N), etc. For details, see the tutorial mentioned above.