{"id":1748,"date":"2017-11-07T07:43:52","date_gmt":"2017-11-07T12:43:52","guid":{"rendered":"http:\/\/titan.chem.uottawa.ca\/?page_id=1748"},"modified":"2017-11-07T07:43:52","modified_gmt":"2017-11-07T12:43:52","slug":"wooki-cluster-job-submission","status":"publish","type":"page","link":"http:\/\/titan.chem.uottawa.ca\/?page_id=1748","title":{"rendered":"Wooki Cluster &#8211; Job Submission"},"content":{"rendered":"<h1 id=\"Job_Submission\">Job Submission<\/h1>\n<p class=\"line874\">Wooki is a shared resource which is essential for the research projects of all group members. To help ensure Wooki&#8217;s smooth operation please read the following set of guidelines.<span id=\"line-5\" class=\"anchor\"><\/span><span id=\"line-6\" class=\"anchor\"><\/span><\/p>\n<h2 id=\"Most_Important_Rules_and_Guidelines\">Most Important Rules and Guidelines<\/h2>\n<ul>\n<li>Always use the queuing system for running jobs.<span id=\"line-9\" class=\"anchor\"><\/span><\/li>\n<li>\n<p class=\"line862\">Use your\u00a0<tt class=\"backtick\">scratch<\/tt>\u00a0directory when running jobs.<span id=\"line-10\" class=\"anchor\"><\/span><\/p>\n<\/li>\n<li>Don&#8217;t store large restart files or intermediate results on your home directory. (It wastes resources during backups)<span id=\"line-11\" class=\"anchor\"><\/span><\/li>\n<li>If you are using a standard package, such as VASP, Gaussian or NAMD check the module help for specific information and use the submit scripts that have been written to submit jobs to the queue. There are submit scripts for each of the most commonly used code packages.<span id=\"line-12\" class=\"anchor\"><\/span><\/li>\n<li>Don&#8217;t run long background or interactive jobs on the frontend. The frontend is reserved for compiling, visualization, data analysis and job setup. If you need more than 15 minutes of CPU time, write a script to submit your job to the queue.<span id=\"line-13\" class=\"anchor\"><\/span><\/li>\n<li>Set the estimated job time so the queue can schedule effectively.<span id=\"line-14\" class=\"anchor\"><\/span><span id=\"line-15\" class=\"anchor\"><\/span><\/li>\n<\/ul>\n<h3 id=\"The_Queuing_System_on_Wooki\">The Queuing System on Wooki<\/h3>\n<p class=\"line874\">The queuing system controls the scheduling, distribution and execution of jobs on the cluster. On Wooki, all compute jobs should be run through the queuing system. Rocks is fully integrated with the Gridengine queue system, also known previously as Sun Grid Engine and Oracle Grid Engine, for historical reasons this document will refer to it as SGE.<span id=\"line-18\" class=\"anchor\"><\/span><span id=\"line-19\" class=\"anchor\"><\/span><\/p>\n<p class=\"line867\"><strong>Basic Commands for Viewing the Status of the Queue<\/strong><span id=\"line-20\" class=\"anchor\"><\/span><span id=\"line-21\" class=\"anchor\"><\/span><\/p>\n<p class=\"line862\">All commands are documented with help by adding the\u00a0<tt class=\"backtick\">-h<\/tt>\u00a0option (i.e.\u00a0<tt class=\"backtick\">qstat\u00a0-h<\/tt>) or using the\u00a0<tt class=\"backtick\">man<\/tt>\u00a0command (i.e.\u00a0<tt class=\"backtick\">man\u00a0qstat<\/tt>).<span id=\"line-22\" class=\"anchor\"><\/span><span id=\"line-23\" class=\"anchor\"><\/span><\/p>\n<p class=\"line867\"><em>qstat<\/em><span id=\"line-24\" class=\"anchor\"><\/span><span id=\"line-25\" class=\"anchor\"><\/span><\/p>\n<p class=\"line874\">Displays the current status of your running and waiting jobs on the queue.<span id=\"line-26\" class=\"anchor\"><\/span><span id=\"line-27\" class=\"anchor\"><\/span><\/p>\n<ul>\n<li>\n<p class=\"line891\"><tt class=\"backtick\">qstat\u00a0-u\u00a0\\*<\/tt>\u00a0shows jobs for all users in priority order.<span id=\"line-28\" class=\"anchor\"><\/span><\/p>\n<\/li>\n<li>\n<p class=\"line891\"><tt class=\"backtick\">qstat\u00a0-f<\/tt>\u00a0shows the nodes that each job is running on.<span id=\"line-29\" class=\"anchor\"><\/span><\/p>\n<\/li>\n<li>\n<p class=\"line891\"><tt class=\"backtick\">qstat\u00a0-g\u00a0c<\/tt>\u00a0shows the number of nodes free in the system<span id=\"line-30\" class=\"anchor\"><\/span><span id=\"line-31\" class=\"anchor\"><\/span><span id=\"line-32\" class=\"anchor\"><\/span><\/p>\n<\/li>\n<\/ul>\n<p class=\"line862\">We are currently in the process of developing some customised scripts to display usage on the cluster in a friendlier manner. For full information or to make requests for functionlaity see the development page:\u00a0<a class=\"https\" href=\"https:\/\/bitbucket.org\/tdaff\/wsge\/\">https:\/\/bitbucket.org\/tdaff\/wsge\/<\/a>\u00a0where you can also open issues for feature requests. Currently commands include:<span id=\"line-33\" class=\"anchor\"><\/span><span id=\"line-34\" class=\"anchor\"><\/span><\/p>\n<ul>\n<li>\n<p class=\"line891\"><strong><tt class=\"backtick\">wstat<\/tt><\/strong>\u00a0gives a more user friendly list of all jobs on the cluster;<span id=\"line-35\" class=\"anchor\"><\/span><\/p>\n<ul>\n<li>\n<p class=\"line891\"><strong><tt class=\"backtick\">wstat\u00a0-u\u00a0username<\/tt><\/strong>\u00a0only shows\u00a0<tt class=\"backtick\">username<\/tt>&#8216;s jobs.<span id=\"line-36\" class=\"anchor\"><\/span><\/p>\n<\/li>\n<\/ul>\n<\/li>\n<li>\n<p class=\"line891\"><strong><tt class=\"backtick\">cluster_stat<\/tt><\/strong>\u00a0shows a comprehensive overview of jobs and users on the cluster.<span id=\"line-37\" class=\"anchor\"><\/span><span id=\"line-38\" class=\"anchor\"><\/span><\/p>\n<\/li>\n<\/ul>\n<h3 id=\"Submitting_Jobs_on_Wooki\">Submitting Jobs on Wooki<\/h3>\n<p class=\"line862\">For most codes that come as a module there will be a\u00a0<strong><tt class=\"backtick\">$code-submit<\/tt><\/strong>\u00a0script that takes care of building an SGE script and submitting the job, for example\u00a0<strong><tt class=\"backtick\">vasp-submit<\/tt><\/strong>\u00a0or\u00a0<strong><tt class=\"backtick\">gaussian-submit<\/tt><\/strong>. Please use these if they are available. These scripts ensure the proper usage of the local scratch disks and for parallel jobs, they set up the parallel execution environments and message passing systems.<span id=\"line-41\" class=\"anchor\"><\/span><span id=\"line-42\" class=\"anchor\"><\/span><\/p>\n<p class=\"line862\">The submit scripts are located with their modules in:\u00a0<em>\/share\/apps<\/em>, but when you\u00a0<em>load the module<\/em>\u00a0they will be available as commands in your shell.<span id=\"line-43\" class=\"anchor\"><\/span><span id=\"line-44\" class=\"anchor\"><\/span><\/p>\n<p class=\"line874\">If a commonly used program is missing a submit script, you may open a ticket to ask for one to be written.<span id=\"line-45\" class=\"anchor\"><\/span><span id=\"line-46\" class=\"anchor\"><\/span><\/p>\n<h3 id=\"Basic_Usage_of_Submit_Scripts\">Basic Usage of Submit Scripts<\/h3>\n<p class=\"line874\">To show the basic usage and behaviour of the submit scripts we will use the example of submitting a vasp job. Most submit scripts operate in a very similar way.<span id=\"line-54\" class=\"anchor\"><\/span><span id=\"line-55\" class=\"anchor\"><\/span><\/p>\n<div class=\"highlight console\">\n<div class=\"codearea\" dir=\"ltr\" lang=\"en\"><a class=\"codenumbers\" href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#\">Toggle line numbers<\/a><\/p>\n<pre id=\"CA-f804ffead9898907b030694d27298edd1e803f3f\" dir=\"ltr\" lang=\"en\"><span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-f804ffead9898907b030694d27298edd1e803f3f_1\">   1<\/a> <\/span><span id=\"CA-f804ffead9898907b030694d27298edd1e803f3f_1\" class=\"LineAnchor\"><\/span><span id=\"line-1-2\" class=\"anchor\"><\/span><span class=\"ID\">$<\/span> module <span class=\"ResWord\">help <\/span>vasp<\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-f804ffead9898907b030694d27298edd1e803f3f_2\">   2<\/a> <\/span><span id=\"CA-f804ffead9898907b030694d27298edd1e803f3f_2\" class=\"LineAnchor\"><\/span><span id=\"line-2-1\" class=\"anchor\"><\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-f804ffead9898907b030694d27298edd1e803f3f_3\">   3<\/a> <\/span><span id=\"CA-f804ffead9898907b030694d27298edd1e803f3f_3\" class=\"LineAnchor\"><\/span><span id=\"line-3-1\" class=\"anchor\"><\/span><span class=\"LineNumber\">----------- Module Specific Help for 'vasp\/5.3.5' -----------------<\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-f804ffead9898907b030694d27298edd1e803f3f_4\">   4<\/a> <\/span><span id=\"CA-f804ffead9898907b030694d27298edd1e803f3f_4\" class=\"LineAnchor\"><\/span><span id=\"line-4-1\" class=\"anchor\"><\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-f804ffead9898907b030694d27298edd1e803f3f_5\">   5<\/a> <\/span><span id=\"CA-f804ffead9898907b030694d27298edd1e803f3f_5\" class=\"LineAnchor\"><\/span><span id=\"line-5-1\" class=\"anchor\"><\/span><span class=\"LineNumber\">    Provides VASP 5.3.5<\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-f804ffead9898907b030694d27298edd1e803f3f_6\">   6<\/a> <\/span><span id=\"CA-f804ffead9898907b030694d27298edd1e803f3f_6\" class=\"LineAnchor\"><\/span><span id=\"line-6-1\" class=\"anchor\"><\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-f804ffead9898907b030694d27298edd1e803f3f_7\">   7<\/a> <\/span><span id=\"CA-f804ffead9898907b030694d27298edd1e803f3f_7\" class=\"LineAnchor\"><\/span><span id=\"line-7-1\" class=\"anchor\"><\/span><span class=\"LineNumber\">    Use `module load vasp` to load the package then<\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-f804ffead9898907b030694d27298edd1e803f3f_8\">   8<\/a> <\/span><span id=\"CA-f804ffead9898907b030694d27298edd1e803f3f_8\" class=\"LineAnchor\"><\/span><span id=\"line-8-1\" class=\"anchor\"><\/span><span class=\"LineNumber\">    use the command `vasp-submit` to run jobs through<\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-f804ffead9898907b030694d27298edd1e803f3f_9\">   9<\/a> <\/span><span id=\"CA-f804ffead9898907b030694d27298edd1e803f3f_9\" class=\"LineAnchor\"><\/span><span id=\"line-9-1\" class=\"anchor\"><\/span><span class=\"LineNumber\">    the queue.<\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-f804ffead9898907b030694d27298edd1e803f3f_10\">  10<\/a> <\/span><span id=\"CA-f804ffead9898907b030694d27298edd1e803f3f_10\" class=\"LineAnchor\"><\/span><span id=\"line-10-1\" class=\"anchor\"><\/span><span class=\"ID\">$<\/span> module load vasp<\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-f804ffead9898907b030694d27298edd1e803f3f_11\">  11<\/a> <\/span><span id=\"CA-f804ffead9898907b030694d27298edd1e803f3f_11\" class=\"LineAnchor\"><\/span><span id=\"line-11-1\" class=\"anchor\"><\/span><span class=\"ID\">$<\/span> vasp-submit --help<\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-f804ffead9898907b030694d27298edd1e803f3f_12\">  12<\/a> <\/span><span id=\"CA-f804ffead9898907b030694d27298edd1e803f3f_12\" class=\"LineAnchor\"><\/span><span id=\"line-12-1\" class=\"anchor\"><\/span><span class=\"LineNumber\">usage: vasp-submit [-h] [-i] [-r RUNTIME] [-t] [-g] [-d] job_name [num_cpu]<\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-f804ffead9898907b030694d27298edd1e803f3f_13\">  13<\/a> <\/span><span id=\"CA-f804ffead9898907b030694d27298edd1e803f3f_13\" class=\"LineAnchor\"><\/span><span id=\"line-13-1\" class=\"anchor\"><\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-f804ffead9898907b030694d27298edd1e803f3f_14\">  14<\/a> <\/span><span id=\"CA-f804ffead9898907b030694d27298edd1e803f3f_14\" class=\"LineAnchor\"><\/span><span id=\"line-14-1\" class=\"anchor\"><\/span><span class=\"LineNumber\">Submit a vasp job.<\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-f804ffead9898907b030694d27298edd1e803f3f_15\">  15<\/a> <\/span><span id=\"CA-f804ffead9898907b030694d27298edd1e803f3f_15\" class=\"LineAnchor\"><\/span><span id=\"line-15-1\" class=\"anchor\"><\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-f804ffead9898907b030694d27298edd1e803f3f_16\">  16<\/a> <\/span><span id=\"CA-f804ffead9898907b030694d27298edd1e803f3f_16\" class=\"LineAnchor\"><\/span><span id=\"line-16-1\" class=\"anchor\"><\/span><span class=\"LineNumber\">positional arguments:<\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-f804ffead9898907b030694d27298edd1e803f3f_17\">  17<\/a> <\/span><span id=\"CA-f804ffead9898907b030694d27298edd1e803f3f_17\" class=\"LineAnchor\"><\/span><span id=\"line-17-1\" class=\"anchor\"><\/span><span class=\"LineNumber\">  job_name              vasp job name<\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-f804ffead9898907b030694d27298edd1e803f3f_18\">  18<\/a> <\/span><span id=\"CA-f804ffead9898907b030694d27298edd1e803f3f_18\" class=\"LineAnchor\"><\/span><span id=\"line-18-1\" class=\"anchor\"><\/span><span class=\"LineNumber\">  num_cpu               number of processosr to run on (default: 1)<\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-f804ffead9898907b030694d27298edd1e803f3f_19\">  19<\/a> <\/span><span id=\"CA-f804ffead9898907b030694d27298edd1e803f3f_19\" class=\"LineAnchor\"><\/span><span id=\"line-19-1\" class=\"anchor\"><\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-f804ffead9898907b030694d27298edd1e803f3f_20\">  20<\/a> <\/span><span id=\"CA-f804ffead9898907b030694d27298edd1e803f3f_20\" class=\"LineAnchor\"><\/span><span id=\"line-20-1\" class=\"anchor\"><\/span><span class=\"LineNumber\">optional arguments:<\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-f804ffead9898907b030694d27298edd1e803f3f_21\">  21<\/a> <\/span><span id=\"CA-f804ffead9898907b030694d27298edd1e803f3f_21\" class=\"LineAnchor\"><\/span><span id=\"line-21-1\" class=\"anchor\"><\/span><span class=\"LineNumber\">  -h, --help            show this help message and exit<\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-f804ffead9898907b030694d27298edd1e803f3f_22\">  22<\/a> <\/span><span id=\"CA-f804ffead9898907b030694d27298edd1e803f3f_22\" class=\"LineAnchor\"><\/span><span id=\"line-22-1\" class=\"anchor\"><\/span><span class=\"LineNumber\">  -i, --infiniband      run on infiniband for large parallel jobs<\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-f804ffead9898907b030694d27298edd1e803f3f_23\">  23<\/a> <\/span><span id=\"CA-f804ffead9898907b030694d27298edd1e803f3f_23\" class=\"LineAnchor\"><\/span><span id=\"line-23-1\" class=\"anchor\"><\/span><span class=\"LineNumber\">  -r RUNTIME, --runtime RUNTIME<\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-f804ffead9898907b030694d27298edd1e803f3f_24\">  24<\/a> <\/span><span id=\"CA-f804ffead9898907b030694d27298edd1e803f3f_24\" class=\"LineAnchor\"><\/span><span id=\"line-24-1\" class=\"anchor\"><\/span><span class=\"LineNumber\">                        maximum length of time for the job to run, specify as<\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-f804ffead9898907b030694d27298edd1e803f3f_25\">  25<\/a> <\/span><span id=\"CA-f804ffead9898907b030694d27298edd1e803f3f_25\" class=\"LineAnchor\"><\/span><span id=\"line-25-1\" class=\"anchor\"><\/span><span class=\"LineNumber\">                        number of seconds or hh:mm:ss<\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-f804ffead9898907b030694d27298edd1e803f3f_26\">  26<\/a> <\/span><span id=\"CA-f804ffead9898907b030694d27298edd1e803f3f_26\" class=\"LineAnchor\"><\/span><span id=\"line-26-1\" class=\"anchor\"><\/span><span class=\"LineNumber\">  -t, --threaded        keep the processes in a single node<\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-f804ffead9898907b030694d27298edd1e803f3f_27\">  27<\/a> <\/span><span id=\"CA-f804ffead9898907b030694d27298edd1e803f3f_27\" class=\"LineAnchor\"><\/span><span id=\"line-27-1\" class=\"anchor\"><\/span><span class=\"LineNumber\">  -g, --gamma           use the gamma point vasp executable<\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-f804ffead9898907b030694d27298edd1e803f3f_28\">  28<\/a> <\/span><span id=\"CA-f804ffead9898907b030694d27298edd1e803f3f_28\" class=\"LineAnchor\"><\/span><span id=\"line-28-1\" class=\"anchor\"><\/span><span class=\"LineNumber\">  -d, --debug           print script rather than submit<\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-f804ffead9898907b030694d27298edd1e803f3f_29\">  29<\/a> <\/span><span id=\"CA-f804ffead9898907b030694d27298edd1e803f3f_29\" class=\"LineAnchor\"><\/span><span id=\"line-29-1\" class=\"anchor\"><\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-f804ffead9898907b030694d27298edd1e803f3f_30\">  30<\/a> <\/span><span id=\"CA-f804ffead9898907b030694d27298edd1e803f3f_30\" class=\"LineAnchor\"><\/span><span id=\"line-30-1\" class=\"anchor\"><\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-f804ffead9898907b030694d27298edd1e803f3f_31\">  31<\/a> <\/span><span id=\"CA-f804ffead9898907b030694d27298edd1e803f3f_31\" class=\"LineAnchor\"><\/span><span id=\"line-31-1\" class=\"anchor\"><\/span><span class=\"ID\">$<\/span> vasp-submit -g -r 24:00:00 surface 16<\/span>\r\n<\/pre>\n<\/div>\n<\/div>\n<ul>\n<li>This submits a gamma point version vasp job on 16 CPUs to run for no more than a day. Be aware that some scripts requre the input file name, rather than just a job name.<span id=\"line-90\" class=\"anchor\"><\/span><\/li>\n<li>\n<p class=\"line862\">You can run the command\u00a0<tt class=\"backtick\">module\u00a0avail<\/tt>\u00a0to give a list of all installed packages.<span id=\"line-91\" class=\"anchor\"><\/span><\/p>\n<\/li>\n<li>\n<p class=\"line862\">Running\u00a0<tt class=\"backtick\">module\u00a0help\u00a0...<\/tt>\u00a0will give you any other important instructions for that software.<span id=\"line-92\" class=\"anchor\"><\/span><\/p>\n<\/li>\n<li>\n<p class=\"line862\">The\u00a0<tt class=\"backtick\">--help<\/tt>\u00a0option of the submission script will describe any extra requirements for those jobs.<span id=\"line-93\" class=\"anchor\"><\/span><\/p>\n<\/li>\n<li>It is recommended that the running time be specified for all jobs so that they can be scheduled effectively. Be aware that the jobs get killed after this time.<span id=\"line-94\" class=\"anchor\"><\/span><span id=\"line-95\" class=\"anchor\"><\/span><\/li>\n<\/ul>\n<p class=\"line862\">Submitting jobs for other software pacakges will be very similar, but there may be some subtle differences. For example\u00a0<tt class=\"backtick\">gaussian-submit<\/tt>\u00a0takes the input filename as an argument and will check that your input file requests the correct number of CPUs.<span id=\"line-96\" class=\"anchor\"><\/span><span id=\"line-97\" class=\"anchor\"><\/span><\/p>\n<div class=\"highlight console\">\n<div class=\"codearea\" dir=\"ltr\" lang=\"en\"><a class=\"codenumbers\" href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#\">Toggle line numbers<\/a><\/p>\n<pre id=\"CA-f79ac61f46b3791743c782ba4a6a523eba6c2170\" dir=\"ltr\" lang=\"en\"><span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-f79ac61f46b3791743c782ba4a6a523eba6c2170_1\">   1<\/a> <\/span><span id=\"CA-f79ac61f46b3791743c782ba4a6a523eba6c2170_1\" class=\"LineAnchor\"><\/span><span id=\"line-1-4\" class=\"anchor\"><\/span><span class=\"ID\">$<\/span> module <span class=\"ResWord\">help <\/span>gaussian<\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-f79ac61f46b3791743c782ba4a6a523eba6c2170_2\">   2<\/a> <\/span><span id=\"CA-f79ac61f46b3791743c782ba4a6a523eba6c2170_2\" class=\"LineAnchor\"><\/span><span id=\"line-2-2\" class=\"anchor\"><\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-f79ac61f46b3791743c782ba4a6a523eba6c2170_3\">   3<\/a> <\/span><span id=\"CA-f79ac61f46b3791743c782ba4a6a523eba6c2170_3\" class=\"LineAnchor\"><\/span><span id=\"line-3-2\" class=\"anchor\"><\/span><span class=\"LineNumber\">----------- Module Specific Help for 'gaussian\/g09_D.01' ----------<\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-f79ac61f46b3791743c782ba4a6a523eba6c2170_4\">   4<\/a> <\/span><span id=\"CA-f79ac61f46b3791743c782ba4a6a523eba6c2170_4\" class=\"LineAnchor\"><\/span><span id=\"line-4-2\" class=\"anchor\"><\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-f79ac61f46b3791743c782ba4a6a523eba6c2170_5\">   5<\/a> <\/span><span id=\"CA-f79ac61f46b3791743c782ba4a6a523eba6c2170_5\" class=\"LineAnchor\"><\/span><span id=\"line-5-2\" class=\"anchor\"><\/span><span class=\"LineNumber\">    Provides Gaussian g09_D.01 for Intel processors<\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-f79ac61f46b3791743c782ba4a6a523eba6c2170_6\">   6<\/a> <\/span><span id=\"CA-f79ac61f46b3791743c782ba4a6a523eba6c2170_6\" class=\"LineAnchor\"><\/span><span id=\"line-6-2\" class=\"anchor\"><\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-f79ac61f46b3791743c782ba4a6a523eba6c2170_7\">   7<\/a> <\/span><span id=\"CA-f79ac61f46b3791743c782ba4a6a523eba6c2170_7\" class=\"LineAnchor\"><\/span><span id=\"line-7-2\" class=\"anchor\"><\/span><span class=\"LineNumber\">    To load the Gaussian package, use this command:<\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-f79ac61f46b3791743c782ba4a6a523eba6c2170_8\">   8<\/a> <\/span><span id=\"CA-f79ac61f46b3791743c782ba4a6a523eba6c2170_8\" class=\"LineAnchor\"><\/span><span id=\"line-8-2\" class=\"anchor\"><\/span><span class=\"LineNumber\">    &gt; module load gaussian<\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-f79ac61f46b3791743c782ba4a6a523eba6c2170_9\">   9<\/a> <\/span><span id=\"CA-f79ac61f46b3791743c782ba4a6a523eba6c2170_9\" class=\"LineAnchor\"><\/span><span id=\"line-9-2\" class=\"anchor\"><\/span><span class=\"LineNumber\">    To run jobs through the queue, use this command:<\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-f79ac61f46b3791743c782ba4a6a523eba6c2170_10\">  10<\/a> <\/span><span id=\"CA-f79ac61f46b3791743c782ba4a6a523eba6c2170_10\" class=\"LineAnchor\"><\/span><span id=\"line-10-2\" class=\"anchor\"><\/span><span class=\"LineNumber\">    &gt; gaussian-submit input_file #CPU<\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-f79ac61f46b3791743c782ba4a6a523eba6c2170_11\">  11<\/a> <\/span><span id=\"CA-f79ac61f46b3791743c782ba4a6a523eba6c2170_11\" class=\"LineAnchor\"><\/span><span id=\"line-11-2\" class=\"anchor\"><\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-f79ac61f46b3791743c782ba4a6a523eba6c2170_12\">  12<\/a> <\/span><span id=\"CA-f79ac61f46b3791743c782ba4a6a523eba6c2170_12\" class=\"LineAnchor\"><\/span><span id=\"line-12-2\" class=\"anchor\"><\/span><span class=\"ID\">$<\/span> module load gaussian<\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-f79ac61f46b3791743c782ba4a6a523eba6c2170_13\">  13<\/a> <\/span><span id=\"CA-f79ac61f46b3791743c782ba4a6a523eba6c2170_13\" class=\"LineAnchor\"><\/span><span id=\"line-13-2\" class=\"anchor\"><\/span><span class=\"ID\">$<\/span> gaussian-submit --help<\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-f79ac61f46b3791743c782ba4a6a523eba6c2170_14\">  14<\/a> <\/span><span id=\"CA-f79ac61f46b3791743c782ba4a6a523eba6c2170_14\" class=\"LineAnchor\"><\/span><span id=\"line-14-2\" class=\"anchor\"><\/span><span class=\"LineNumber\">usage: gaussian-submit [-h] [-r RUNTIME] [-d] input_file [num_cpu]<\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-f79ac61f46b3791743c782ba4a6a523eba6c2170_15\">  15<\/a> <\/span><span id=\"CA-f79ac61f46b3791743c782ba4a6a523eba6c2170_15\" class=\"LineAnchor\"><\/span><span id=\"line-15-2\" class=\"anchor\"><\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-f79ac61f46b3791743c782ba4a6a523eba6c2170_16\">  16<\/a> <\/span><span id=\"CA-f79ac61f46b3791743c782ba4a6a523eba6c2170_16\" class=\"LineAnchor\"><\/span><span id=\"line-16-2\" class=\"anchor\"><\/span><span class=\"LineNumber\">Submit a gaussian job.<\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-f79ac61f46b3791743c782ba4a6a523eba6c2170_17\">  17<\/a> <\/span><span id=\"CA-f79ac61f46b3791743c782ba4a6a523eba6c2170_17\" class=\"LineAnchor\"><\/span><span id=\"line-17-2\" class=\"anchor\"><\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-f79ac61f46b3791743c782ba4a6a523eba6c2170_18\">  18<\/a> <\/span><span id=\"CA-f79ac61f46b3791743c782ba4a6a523eba6c2170_18\" class=\"LineAnchor\"><\/span><span id=\"line-18-2\" class=\"anchor\"><\/span><span class=\"LineNumber\">positional arguments:<\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-f79ac61f46b3791743c782ba4a6a523eba6c2170_19\">  19<\/a> <\/span><span id=\"CA-f79ac61f46b3791743c782ba4a6a523eba6c2170_19\" class=\"LineAnchor\"><\/span><span id=\"line-19-2\" class=\"anchor\"><\/span><span class=\"LineNumber\">  input_file            gaussian input file name<\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-f79ac61f46b3791743c782ba4a6a523eba6c2170_20\">  20<\/a> <\/span><span id=\"CA-f79ac61f46b3791743c782ba4a6a523eba6c2170_20\" class=\"LineAnchor\"><\/span><span id=\"line-20-2\" class=\"anchor\"><\/span><span class=\"LineNumber\">  num_cpu               number of processosr to run on (default: 1)<\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-f79ac61f46b3791743c782ba4a6a523eba6c2170_21\">  21<\/a> <\/span><span id=\"CA-f79ac61f46b3791743c782ba4a6a523eba6c2170_21\" class=\"LineAnchor\"><\/span><span id=\"line-21-2\" class=\"anchor\"><\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-f79ac61f46b3791743c782ba4a6a523eba6c2170_22\">  22<\/a> <\/span><span id=\"CA-f79ac61f46b3791743c782ba4a6a523eba6c2170_22\" class=\"LineAnchor\"><\/span><span id=\"line-22-2\" class=\"anchor\"><\/span><span class=\"LineNumber\">optional arguments:<\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-f79ac61f46b3791743c782ba4a6a523eba6c2170_23\">  23<\/a> <\/span><span id=\"CA-f79ac61f46b3791743c782ba4a6a523eba6c2170_23\" class=\"LineAnchor\"><\/span><span id=\"line-23-2\" class=\"anchor\"><\/span><span class=\"LineNumber\">  -h, --help            show this help message and exit<\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-f79ac61f46b3791743c782ba4a6a523eba6c2170_24\">  24<\/a> <\/span><span id=\"CA-f79ac61f46b3791743c782ba4a6a523eba6c2170_24\" class=\"LineAnchor\"><\/span><span id=\"line-24-2\" class=\"anchor\"><\/span><span class=\"LineNumber\">  -r RUNTIME, --runtime RUNTIME<\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-f79ac61f46b3791743c782ba4a6a523eba6c2170_25\">  25<\/a> <\/span><span id=\"CA-f79ac61f46b3791743c782ba4a6a523eba6c2170_25\" class=\"LineAnchor\"><\/span><span id=\"line-25-2\" class=\"anchor\"><\/span><span class=\"LineNumber\">                        maximum length of time for the job to run, specify as<\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-f79ac61f46b3791743c782ba4a6a523eba6c2170_26\">  26<\/a> <\/span><span id=\"CA-f79ac61f46b3791743c782ba4a6a523eba6c2170_26\" class=\"LineAnchor\"><\/span><span id=\"line-26-2\" class=\"anchor\"><\/span><span class=\"LineNumber\">                        number of seconds or hh:mm:ss<\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-f79ac61f46b3791743c782ba4a6a523eba6c2170_27\">  27<\/a> <\/span><span id=\"CA-f79ac61f46b3791743c782ba4a6a523eba6c2170_27\" class=\"LineAnchor\"><\/span><span id=\"line-27-2\" class=\"anchor\"><\/span><span class=\"LineNumber\">  -d, --debug           print script rather than submit<\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-f79ac61f46b3791743c782ba4a6a523eba6c2170_28\">  28<\/a> <\/span><span id=\"CA-f79ac61f46b3791743c782ba4a6a523eba6c2170_28\" class=\"LineAnchor\"><\/span><span id=\"line-28-2\" class=\"anchor\"><\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-f79ac61f46b3791743c782ba4a6a523eba6c2170_29\">  29<\/a> <\/span><span id=\"CA-f79ac61f46b3791743c782ba4a6a523eba6c2170_29\" class=\"LineAnchor\"><\/span><span id=\"line-29-2\" class=\"anchor\"><\/span><span class=\"ID\">$<\/span> gaussian-submit benzene.gjf 2<\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-f79ac61f46b3791743c782ba4a6a523eba6c2170_30\">  30<\/a> <\/span><span id=\"CA-f79ac61f46b3791743c782ba4a6a523eba6c2170_30\" class=\"LineAnchor\"><\/span><span id=\"line-30-2\" class=\"anchor\"><\/span><span class=\"LineNumber\">Input file requests too many CPUs (6), not submitting<\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-f79ac61f46b3791743c782ba4a6a523eba6c2170_31\">  31<\/a> <\/span><span id=\"CA-f79ac61f46b3791743c782ba4a6a523eba6c2170_31\" class=\"LineAnchor\"><\/span><span id=\"line-31-2\" class=\"anchor\"><\/span><span class=\"ID\">$<\/span> gaussian-submit benzene.gjf 6<\/span>\r\n<\/pre>\n<\/div>\n<\/div>\n<h3 id=\"Output_Files_and_Log_Files\">Output Files and Log Files<\/h3>\n<p class=\"line874\">The output and\/or logfiles will be written to a file in the directory where the job was submitted. The files will be called &#8220;jobname.out&#8221; or &#8220;jobname.log&#8221;. For example, if the input file that was submitted was called &#8220;pd_test.in&#8221;, then the output file would be &#8220;pd_test.out&#8221;. The output and\/or log files can be viewed during job execution from the headnode to examine the progress of the simulation.<span id=\"line-134\" class=\"anchor\"><\/span><span id=\"line-135\" class=\"anchor\"><\/span><\/p>\n<p class=\"line874\">Terminating a Job on the Queue<span id=\"line-136\" class=\"anchor\"><\/span><span id=\"line-137\" class=\"anchor\"><\/span><\/p>\n<p class=\"line862\">Use the\u00a0<strong>qdel<\/strong>\u00a0command to terminate running or waiting jobs on the queue. For example:<span id=\"line-138\" class=\"anchor\"><\/span><span id=\"line-139\" class=\"anchor\"><\/span><\/p>\n<div class=\"highlight console\">\n<div class=\"codearea\" dir=\"ltr\" lang=\"en\"><a class=\"codenumbers\" href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#\">Toggle line numbers<\/a><\/p>\n<pre id=\"CA-d940232f26c3c633f52ebd4c2d1a400d900c5024\" dir=\"ltr\" lang=\"en\"><span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-d940232f26c3c633f52ebd4c2d1a400d900c5024_1\">   1<\/a> <\/span><span id=\"CA-d940232f26c3c633f52ebd4c2d1a400d900c5024_1\" class=\"LineAnchor\"><\/span><span id=\"line-1-6\" class=\"anchor\"><\/span><span class=\"ID\">$<\/span> qstat<\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-d940232f26c3c633f52ebd4c2d1a400d900c5024_2\">   2<\/a> <\/span><span id=\"CA-d940232f26c3c633f52ebd4c2d1a400d900c5024_2\" class=\"LineAnchor\"><\/span><span id=\"line-2-3\" class=\"anchor\"><\/span><span class=\"LineNumber\">qstat<\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-d940232f26c3c633f52ebd4c2d1a400d900c5024_3\">   3<\/a> <\/span><span id=\"CA-d940232f26c3c633f52ebd4c2d1a400d900c5024_3\" class=\"LineAnchor\"><\/span><span id=\"line-3-3\" class=\"anchor\"><\/span><span class=\"LineNumber\">job-ID  prior   name       user         state submit\/start at     queue                          slots ja-task-ID <\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-d940232f26c3c633f52ebd4c2d1a400d900c5024_4\">   4<\/a> <\/span><span id=\"CA-d940232f26c3c633f52ebd4c2d1a400d900c5024_4\" class=\"LineAnchor\"><\/span><span id=\"line-4-3\" class=\"anchor\"><\/span><span class=\"LineNumber\">-----------------------------------------------------------------------------------------------------------------<\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-d940232f26c3c633f52ebd4c2d1a400d900c5024_5\">   5<\/a> <\/span><span id=\"CA-d940232f26c3c633f52ebd4c2d1a400d900c5024_5\" class=\"LineAnchor\"><\/span><span id=\"line-5-3\" class=\"anchor\"><\/span><span class=\"LineNumber\">1200305 0.62499 vasp.N2_SI tdaff        r     09\/30\/2013 13:37:50 all.q@compute-7-26.local          18        <\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-d940232f26c3c633f52ebd4c2d1a400d900c5024_6\">   6<\/a> <\/span><span id=\"CA-d940232f26c3c633f52ebd4c2d1a400d900c5024_6\" class=\"LineAnchor\"><\/span><span id=\"line-6-3\" class=\"anchor\"><\/span><span class=\"LineNumber\">1207188 0.62499 gulp.C1Fre tdaff        r     10\/02\/2013 22:42:22 all.q@compute-7-17.local           1        <\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-d940232f26c3c633f52ebd4c2d1a400d900c5024_7\">   7<\/a> <\/span><span id=\"CA-d940232f26c3c633f52ebd4c2d1a400d900c5024_7\" class=\"LineAnchor\"><\/span><span id=\"line-7-3\" class=\"anchor\"><\/span><span class=\"LineNumber\">1209015 0.62499 fastmc.C3F tdaff        r     10\/02\/2013 23:18:22 all.q@compute-1-4.local            1        <\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-d940232f26c3c633f52ebd4c2d1a400d900c5024_8\">   8<\/a> <\/span><span id=\"CA-d940232f26c3c633f52ebd4c2d1a400d900c5024_8\" class=\"LineAnchor\"><\/span><span id=\"line-8-3\" class=\"anchor\"><\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-d940232f26c3c633f52ebd4c2d1a400d900c5024_9\">   9<\/a> <\/span><span id=\"CA-d940232f26c3c633f52ebd4c2d1a400d900c5024_9\" class=\"LineAnchor\"><\/span><span id=\"line-9-3\" class=\"anchor\"><\/span><span class=\"ID\">$<\/span> qdel 1200305<\/span>\r\n<\/pre>\n<\/div>\n<\/div>\n<p class=\"line874\">When a job is terminated in this fashion, it is not a &#8216;clean&#8217; termination of the job and the working directory is left on the execution node in its scratch directory.<span id=\"line-154\" class=\"anchor\"><\/span><span id=\"line-155\" class=\"anchor\"><\/span><\/p>\n<p class=\"line862\">Occasionally, a job will not leave the queue even with repeated qdel commands issued. If a job is stubborn, you can use\u00a0<tt class=\"backtick\">qdel\u00a0-f<\/tt>to force the deletion (takes a couple of minutes). If this does not work one of the nodes that the job was running on has likely crashed. Please open a ticket to report the problem.<span id=\"line-156\" class=\"anchor\"><\/span><span id=\"line-157\" class=\"anchor\"><\/span><\/p>\n<h3 id=\"Location_of_Restart_Files_After_Job_Completion\">Location of Restart Files After Job Completion<\/h3>\n<p class=\"line862\">By default, upon job completion the script will copy the contents of the execution directory back to the submission directory, if this has not occurred there will be a file\u00a0<tt class=\"backtick\">where_is_my_job<\/tt>\u00a0that has commands to take you to the files:<span id=\"line-160\" class=\"anchor\"><\/span><span id=\"line-161\" class=\"anchor\"><\/span><\/p>\n<pre><span id=\"line-1\" class=\"anchor\"><\/span>$ ls\r\n<span id=\"line-2\" class=\"anchor\"><\/span>input.in   output.out   where_is_my_job\r\n<span id=\"line-3\" class=\"anchor\"><\/span>$ cat where_is_my_job\r\n<span id=\"line-4\" class=\"anchor\"><\/span>ssh compute-5-12\r\n<span id=\"line-5\" class=\"anchor\"><\/span>cd \/state\/partition1\/tdaff\/1209015\r\n<span id=\"line-6\" class=\"anchor\"><\/span>$ ssh compute-5-12\r\n<span id=\"line-7\" class=\"anchor\"><\/span>$ cd \/state\/partition1\/tdaff\/1209015\r\n<span id=\"line-8\" class=\"anchor\"><\/span>$ ls\r\n<span id=\"line-9\" class=\"anchor\"><\/span>input.in  restart.file  intermediate.bin<\/pre>\n<h3 id=\"Location_of_the_Execution_Directory\">Location of the Execution Directory<\/h3>\n<p class=\"line862\">Jobs will usually run from the directory they are submitted (know as the\u00a0<tt class=\"backtick\">$SGE_O_WORKDIR<\/tt>). By running jobs from the scratch drives, less load will be placed on the frontend drives to keep it responsive for interactive users. If you have trouble identifying your jobs you can track extended information from the\u00a0<tt class=\"backtick\">qstat\u00a0-j<\/tt>\u00a0command:<span id=\"line-177\" class=\"anchor\"><\/span><span id=\"line-178\" class=\"anchor\"><\/span><\/p>\n<pre><span id=\"line-1-1\" class=\"anchor\"><\/span>$ qstat -j 3233675\r\n<span id=\"line-2-1\" class=\"anchor\"><\/span>==============================================================\r\n<span id=\"line-3-1\" class=\"anchor\"><\/span>job_number:                 3233675\r\n<span id=\"line-4-1\" class=\"anchor\"><\/span>exec_file:                  job_scripts\/3233675\r\n<span id=\"line-5-1\" class=\"anchor\"><\/span>submission_time:            Fri Apr 11 11:44:45 2014\r\n<span id=\"line-6-1\" class=\"anchor\"><\/span>...\r\n<span id=\"line-7-1\" class=\"anchor\"><\/span>sge_o_shell:                \/bin\/bash\r\n<span id=\"line-8-1\" class=\"anchor\"><\/span>sge_o_workdir:              \/share\/scratch\/tdaff\/simulations\/production\/structure_17\r\n<span id=\"line-9-1\" class=\"anchor\"><\/span>sge_o_host:                 wooki\r\n<span id=\"line-10\" class=\"anchor\"><\/span>...<\/pre>\n<p class=\"line862\">For jobs with heavy file operations, the job should be set up to run from the local drive of the node the jobs runs on. If your code has a well made submission script, a file called\u00a0<tt class=\"backtick\">where_is_my_job<\/tt>\u00a0will be created which gives commands to find the directory that the job is running from.<span id=\"line-194\" class=\"anchor\"><\/span><span id=\"line-195\" class=\"anchor\"><\/span><span id=\"line-196\" class=\"anchor\"><\/span><span id=\"line-197\" class=\"anchor\"><\/span><span id=\"line-198\" class=\"anchor\"><\/span><\/p>\n<h2 id=\"SGE_Scripts\">SGE Scripts<\/h2>\n<p class=\"line874\">The script generated by the sumission command has the elements detailed below:<span id=\"line-201\" class=\"anchor\"><\/span><span id=\"line-202\" class=\"anchor\"><\/span><\/p>\n<div class=\"highlight sh\">\n<div class=\"codearea\" dir=\"ltr\" lang=\"en\"><a class=\"codenumbers\" href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#\">Toggle line numbers<\/a><\/p>\n<pre id=\"CA-2d6a5b9f14aca03513b646b59825883902d2cf16\" dir=\"ltr\" lang=\"en\"><span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-2d6a5b9f14aca03513b646b59825883902d2cf16_1\">   1<\/a> <\/span><span id=\"CA-2d6a5b9f14aca03513b646b59825883902d2cf16_1\" class=\"LineAnchor\"><\/span><span id=\"line-1-8\" class=\"anchor\"><\/span><span class=\"Comment\">#!\/bin\/bash<\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-2d6a5b9f14aca03513b646b59825883902d2cf16_2\">   2<\/a> <\/span><span id=\"CA-2d6a5b9f14aca03513b646b59825883902d2cf16_2\" class=\"LineAnchor\"><\/span><span id=\"line-2-4\" class=\"anchor\"><\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-2d6a5b9f14aca03513b646b59825883902d2cf16_3\">   3<\/a> <\/span><span id=\"CA-2d6a5b9f14aca03513b646b59825883902d2cf16_3\" class=\"LineAnchor\"><\/span><span id=\"line-3-4\" class=\"anchor\"><\/span><span class=\"Comment\">### Execute where the job was submitted<\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-2d6a5b9f14aca03513b646b59825883902d2cf16_4\">   4<\/a> <\/span><span id=\"CA-2d6a5b9f14aca03513b646b59825883902d2cf16_4\" class=\"LineAnchor\"><\/span><span id=\"line-4-4\" class=\"anchor\"><\/span><span class=\"Comment\">#$ -cwd<\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-2d6a5b9f14aca03513b646b59825883902d2cf16_5\">   5<\/a> <\/span><span id=\"CA-2d6a5b9f14aca03513b646b59825883902d2cf16_5\" class=\"LineAnchor\"><\/span><span id=\"line-5-4\" class=\"anchor\"><\/span><span class=\"Comment\">### Carry over current environment<\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-2d6a5b9f14aca03513b646b59825883902d2cf16_6\">   6<\/a> <\/span><span id=\"CA-2d6a5b9f14aca03513b646b59825883902d2cf16_6\" class=\"LineAnchor\"><\/span><span id=\"line-6-4\" class=\"anchor\"><\/span><span class=\"Comment\">#$ -V<\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-2d6a5b9f14aca03513b646b59825883902d2cf16_7\">   7<\/a> <\/span><span id=\"CA-2d6a5b9f14aca03513b646b59825883902d2cf16_7\" class=\"LineAnchor\"><\/span><span id=\"line-7-4\" class=\"anchor\"><\/span><span class=\"Comment\">### Put output and error in the same file<\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-2d6a5b9f14aca03513b646b59825883902d2cf16_8\">   8<\/a> <\/span><span id=\"CA-2d6a5b9f14aca03513b646b59825883902d2cf16_8\" class=\"LineAnchor\"><\/span><span id=\"line-8-4\" class=\"anchor\"><\/span><span class=\"Comment\">#$ -j y<\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-2d6a5b9f14aca03513b646b59825883902d2cf16_9\">   9<\/a> <\/span><span id=\"CA-2d6a5b9f14aca03513b646b59825883902d2cf16_9\" class=\"LineAnchor\"><\/span><span id=\"line-9-4\" class=\"anchor\"><\/span><span class=\"Comment\">### Name the output<\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-2d6a5b9f14aca03513b646b59825883902d2cf16_10\">  10<\/a> <\/span><span id=\"CA-2d6a5b9f14aca03513b646b59825883902d2cf16_10\" class=\"LineAnchor\"><\/span><span id=\"line-10-3\" class=\"anchor\"><\/span><span class=\"Comment\">#$ -o jobname.stdout<\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-2d6a5b9f14aca03513b646b59825883902d2cf16_11\">  11<\/a> <\/span><span id=\"CA-2d6a5b9f14aca03513b646b59825883902d2cf16_11\" class=\"LineAnchor\"><\/span><span id=\"line-11-3\" class=\"anchor\"><\/span><span class=\"Comment\">### Hard limit on wallclock time<\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-2d6a5b9f14aca03513b646b59825883902d2cf16_12\">  12<\/a> <\/span><span id=\"CA-2d6a5b9f14aca03513b646b59825883902d2cf16_12\" class=\"LineAnchor\"><\/span><span id=\"line-12-3\" class=\"anchor\"><\/span><span class=\"Comment\">#$ -l h_rt=168:00:00<\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-2d6a5b9f14aca03513b646b59825883902d2cf16_13\">  13<\/a> <\/span><span id=\"CA-2d6a5b9f14aca03513b646b59825883902d2cf16_13\" class=\"LineAnchor\"><\/span><span id=\"line-13-3\" class=\"anchor\"><\/span><span class=\"Comment\">### Make a reservation of nodes<\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-2d6a5b9f14aca03513b646b59825883902d2cf16_14\">  14<\/a> <\/span><span id=\"CA-2d6a5b9f14aca03513b646b59825883902d2cf16_14\" class=\"LineAnchor\"><\/span><span id=\"line-14-3\" class=\"anchor\"><\/span><span class=\"Comment\">#$ -R y<\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-2d6a5b9f14aca03513b646b59825883902d2cf16_15\">  15<\/a> <\/span><span id=\"CA-2d6a5b9f14aca03513b646b59825883902d2cf16_15\" class=\"LineAnchor\"><\/span><span id=\"line-15-3\" class=\"anchor\"><\/span><span class=\"Comment\">### Job name in the qstat list<\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-2d6a5b9f14aca03513b646b59825883902d2cf16_16\">  16<\/a> <\/span><span id=\"CA-2d6a5b9f14aca03513b646b59825883902d2cf16_16\" class=\"LineAnchor\"><\/span><span id=\"line-16-3\" class=\"anchor\"><\/span><span class=\"Comment\">#$ -N jobname<\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-2d6a5b9f14aca03513b646b59825883902d2cf16_17\">  17<\/a> <\/span><span id=\"CA-2d6a5b9f14aca03513b646b59825883902d2cf16_17\" class=\"LineAnchor\"><\/span><span id=\"line-17-3\" class=\"anchor\"><\/span><span class=\"Comment\">### Choose an appropriate parallel environment<\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-2d6a5b9f14aca03513b646b59825883902d2cf16_18\">  18<\/a> <\/span><span id=\"CA-2d6a5b9f14aca03513b646b59825883902d2cf16_18\" class=\"LineAnchor\"><\/span><span id=\"line-18-3\" class=\"anchor\"><\/span><span class=\"Comment\">#$ -pe ppn12_* 48<\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-2d6a5b9f14aca03513b646b59825883902d2cf16_19\">  19<\/a> <\/span><span id=\"CA-2d6a5b9f14aca03513b646b59825883902d2cf16_19\" class=\"LineAnchor\"><\/span><span id=\"line-19-3\" class=\"anchor\"><\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-2d6a5b9f14aca03513b646b59825883902d2cf16_20\">  20<\/a> <\/span><span id=\"CA-2d6a5b9f14aca03513b646b59825883902d2cf16_20\" class=\"LineAnchor\"><\/span><span id=\"line-20-3\" class=\"anchor\"><\/span><span class=\"ResWord\">export <\/span><span class=\"ID\">OMP_NUM_THREADS<\/span>=1<\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-2d6a5b9f14aca03513b646b59825883902d2cf16_21\">  21<\/a> <\/span><span id=\"CA-2d6a5b9f14aca03513b646b59825883902d2cf16_21\" class=\"LineAnchor\"><\/span><span id=\"line-21-3\" class=\"anchor\"><\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-2d6a5b9f14aca03513b646b59825883902d2cf16_22\">  22<\/a> <\/span><span id=\"CA-2d6a5b9f14aca03513b646b59825883902d2cf16_22\" class=\"LineAnchor\"><\/span><span id=\"line-22-3\" class=\"anchor\"><\/span><span class=\"ResWord\">ulimit<\/span> -s unlimited<\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-2d6a5b9f14aca03513b646b59825883902d2cf16_23\">  23<\/a> <\/span><span id=\"CA-2d6a5b9f14aca03513b646b59825883902d2cf16_23\" class=\"LineAnchor\"><\/span><span id=\"line-23-3\" class=\"anchor\"><\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-2d6a5b9f14aca03513b646b59825883902d2cf16_24\">  24<\/a> <\/span><span id=\"CA-2d6a5b9f14aca03513b646b59825883902d2cf16_24\" class=\"LineAnchor\"><\/span><span id=\"line-24-3\" class=\"anchor\"><\/span><span class=\"ResWord\">echo<\/span> <span class=\"String\">\"Job ID $JOB_ID \"<\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-2d6a5b9f14aca03513b646b59825883902d2cf16_25\">  25<\/a> <\/span><span id=\"CA-2d6a5b9f14aca03513b646b59825883902d2cf16_25\" class=\"LineAnchor\"><\/span><span id=\"line-25-3\" class=\"anchor\"><\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-2d6a5b9f14aca03513b646b59825883902d2cf16_26\">  26<\/a> <\/span><span id=\"CA-2d6a5b9f14aca03513b646b59825883902d2cf16_26\" class=\"LineAnchor\"><\/span><span id=\"line-26-3\" class=\"anchor\"><\/span><span class=\"ResWord\">echo<\/span> <span class=\"String\">\"Running on hosts: \"<\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-2d6a5b9f14aca03513b646b59825883902d2cf16_27\">  27<\/a> <\/span><span id=\"CA-2d6a5b9f14aca03513b646b59825883902d2cf16_27\" class=\"LineAnchor\"><\/span><span id=\"line-27-3\" class=\"anchor\"><\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-2d6a5b9f14aca03513b646b59825883902d2cf16_28\">  28<\/a> <\/span><span id=\"CA-2d6a5b9f14aca03513b646b59825883902d2cf16_28\" class=\"LineAnchor\"><\/span><span id=\"line-28-3\" class=\"anchor\"><\/span>cat <span class=\"ID\">$PE_HOSTFILE<\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-2d6a5b9f14aca03513b646b59825883902d2cf16_29\">  29<\/a> <\/span><span id=\"CA-2d6a5b9f14aca03513b646b59825883902d2cf16_29\" class=\"LineAnchor\"><\/span><span id=\"line-29-3\" class=\"anchor\"><\/span><\/span>\r\n<span class=\"line\"><span class=\"LineNumber\"><a href=\"http:\/\/titan.chem.uottawa.ca\/wiki\/Wooki\/JobSubmission#CA-2d6a5b9f14aca03513b646b59825883902d2cf16_30\">  30<\/a> <\/span><span id=\"CA-2d6a5b9f14aca03513b646b59825883902d2cf16_30\" class=\"LineAnchor\"><\/span><span id=\"line-30-3\" class=\"anchor\"><\/span>mpirun -np <span class=\"ID\">$NSLOTS<\/span> \/path\/to\/my\/exe &gt;&gt; job.out<\/span>\r\n<\/pre>\n<\/div>\n<\/div>\n<h2 id=\"Parallel_environments\">Parallel environments<\/h2>\n<p class=\"line874\">Parallel environments define the type of nodes that your job will run on. They are grouped so that jobs do not span different node types or switches and you can select which to use with wildcards (like &#8216;*&#8217;) in your job submission script. A set of environments with set numbers of processes per nodes are defined:<span id=\"line-238\" class=\"anchor\"><\/span><span id=\"line-239\" class=\"anchor\"><\/span><\/p>\n<ul>\n<li>\n<p class=\"line891\"><tt class=\"backtick\">ppn4<\/tt><span id=\"line-240\" class=\"anchor\"><\/span><\/p>\n<\/li>\n<li>\n<p class=\"line891\"><tt class=\"backtick\">ppn6<\/tt><span id=\"line-241\" class=\"anchor\"><\/span><\/p>\n<\/li>\n<li>\n<p class=\"line891\"><tt class=\"backtick\">ppn8<\/tt><span id=\"line-242\" class=\"anchor\"><\/span><\/p>\n<\/li>\n<li>\n<p class=\"line891\"><tt class=\"backtick\">ppn12<\/tt><span id=\"line-243\" class=\"anchor\"><\/span><span id=\"line-244\" class=\"anchor\"><\/span><\/p>\n<\/li>\n<\/ul>\n<p class=\"line874\">with different node types<span id=\"line-245\" class=\"anchor\"><\/span><\/p>\n<ul>\n<li>\n<p class=\"line891\"><tt class=\"backtick\">opteron-8220<\/tt><span id=\"line-246\" class=\"anchor\"><\/span><\/p>\n<\/li>\n<li>\n<p class=\"line891\"><tt class=\"backtick\">xeon-ibm<\/tt><span id=\"line-247\" class=\"anchor\"><\/span><\/p>\n<\/li>\n<li>\n<p class=\"line891\"><tt class=\"backtick\">gpu-m2075<\/tt><span id=\"line-248\" class=\"anchor\"><\/span><\/p>\n<\/li>\n<li>\n<p class=\"line891\"><tt class=\"backtick\">xeon-e5645<\/tt><span id=\"line-249\" class=\"anchor\"><\/span><\/p>\n<\/li>\n<li>\n<p class=\"line891\"><tt class=\"backtick\">xeon-x5650<\/tt><span id=\"line-250\" class=\"anchor\"><\/span><span id=\"line-251\" class=\"anchor\"><\/span><\/p>\n<\/li>\n<\/ul>\n<p class=\"line862\">In most cases it is desirable to reduce spanning and use a high ppn and let the scheduler find the nodes for you. For example, to request 16 cores with 8 processes per node include\u00a0<tt class=\"backtick\"><\/tt><tt class=\"backtick\">#$\u00a0-pe\u00a0ppn8_*\u00a016<\/tt><tt class=\"backtick\"><\/tt>\u00a0in my submission script. Or to run only on Xeons with any number of processes per node, use\u00a0<tt class=\"backtick\"><\/tt><tt class=\"backtick\">#$\u00a0-pe\u00a0ppn*_xeon*\u00a016<\/tt><tt class=\"backtick\"><\/tt>. There are a few special cases:<span id=\"line-252\" class=\"anchor\"><\/span><span id=\"line-253\" class=\"anchor\"><\/span><\/p>\n<ul>\n<li>\n<p class=\"line891\"><tt class=\"backtick\">threaded<\/tt>: Make all the processes run on the same node. The largest node has 16 cores, most have 12 or less. This will give best performance if you can use it.<span id=\"line-254\" class=\"anchor\"><\/span><\/p>\n<\/li>\n<li>\n<p class=\"line891\"><tt class=\"backtick\">ppn4_infiniband*<\/tt>: Use the Infiniband cluster; these nodes only have 4cores but faster interconnects for better scaling of larger jobs.<span id=\"line-255\" class=\"anchor\"><\/span><\/p>\n<\/li>\n<li>\n<p class=\"line891\"><tt class=\"backtick\">orte<\/tt>: Spread processes all over the cluster. Not recommended!<span id=\"line-256\" class=\"anchor\"><\/span><span id=\"line-257\" class=\"anchor\"><\/span><\/p>\n<\/li>\n<\/ul>\n<h2 id=\"SGE_scripts\">SGE scripts<\/h2>\n<p class=\"line862\">Sometimes you need to submit a job onto a node that requires a large amount of memory. Say you need to submit a job that requires 24 GB of memory but the basic\u00a0<strong><tt class=\"backtick\">qsub<\/tt><\/strong>\u00a0command keeps sending the job to a node and returning memory allocation errors. If you use\u00a0<strong><tt class=\"backtick\">qsub\u00a0-l\u00a0mem_free=24G<\/tt><\/strong>\u00a0your job will be sent off to a node that has a minimum of 24GB of free memory.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Job Submission Wooki is a shared resource which is essential for the research projects of all group members. To help ensure Wooki&#8217;s smooth operation please read the following set of guidelines. Most Important Rules and Guidelines Always use the queuing system for running jobs. Use your\u00a0scratch\u00a0directory when running jobs. Don&#8217;t store large restart files or [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-1748","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"http:\/\/titan.chem.uottawa.ca\/index.php?rest_route=\/wp\/v2\/pages\/1748","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/titan.chem.uottawa.ca\/index.php?rest_route=\/wp\/v2\/pages"}],"about":[{"href":"http:\/\/titan.chem.uottawa.ca\/index.php?rest_route=\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"http:\/\/titan.chem.uottawa.ca\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/titan.chem.uottawa.ca\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1748"}],"version-history":[{"count":1,"href":"http:\/\/titan.chem.uottawa.ca\/index.php?rest_route=\/wp\/v2\/pages\/1748\/revisions"}],"predecessor-version":[{"id":1750,"href":"http:\/\/titan.chem.uottawa.ca\/index.php?rest_route=\/wp\/v2\/pages\/1748\/revisions\/1750"}],"wp:attachment":[{"href":"http:\/\/titan.chem.uottawa.ca\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1748"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}