Clubmask User Guide

A node is a machine. The machine may have multiple processors, differnet network adapters, and varying amounts of ram and diskspace, but all in all, it is just a machine. Simple eh?

A job is a basic 'unit of work'. Jobs typically consist of common computations grouped together.

The JOBID is the numerical ID that was assigned to your job at submission time. EX: 1015142526.538906

NODES: space delimited string of node numbers that your job has been assigned. The NODES variable is how you determine which nodes of the cluster your job will occupy. There is no guarantee that NODES will contain contiguous nodes, as the nodes may be scattered across the cluster. Example: NODES="0 0 5 5 2 2 9 9"

Note: If your job has been assigned a dual cpu machine, the node number for it will appear twice, similarily for nodes with quad cpus, it will appear 4 times.

NUMNODES is the number of nodes in NODES. If NODES="1 1 2 2 3 3", NUMNODES will be 6.

Clubmask uses a distributed process control layer called bproc. Bproc allows you to see all of the processes running on remote nodes as if they were running on the main login server. To start programs on nodes using Bproc, you will need to use the command 'bpsh'. In Clubmask, the nodes are numbered starting at 0 and increase as per your setup. Bpsh uses that node number when executing a process.

[henken@alpha henken]$ bpsh 2 hostname
node2
        

[root@struggles clubmask]# bpsh 1 sleep 5
            

[root@struggles autoconf-2.57]# ps -axf
  PID TTY      STAT   TIME COMMAND
  5289 pts/9    S      0:00  bpsh 1 sleep 5
  5291 pts/9    SW     0:00  \_ [sleep]
            

Ssh is the standard secure shell for linux. As stated above, the nodes are numbered starting at 0, but for ssh to correctly resolve the nodenumber to a host, you must prefix the word 'node' to the beginning of the nodenumber.

[henken@alpha henken]$ ssh node2 hostname
node2
        

There are several tradeoffs between bpsh and ssh. First and foremost is the difference in the time in which each takes to start executing the process on the node. Bpsh is several times faster than ssh, and it supports complex node list arguments that allow the processes to be started in near parallel fashion.

Example

[henken@testcluster henken]$ time bpsh 1-5,8,19-31 hostname node1.internal node2.internal node3.internal node4.internal node5.internal node8.internal node19.internal node20.internal node21.internal node22.internal node23.internal node24.internal node25.internal node26.internal node27.internal node28.internal node29.internal node30.internal node31.internal real 0m0.082s user 0m0.040s sys 0m0.170s

To achieve the same thing via ssh you would need to invoke a separate ssh for each node.

Example

[henken@testcluster henken]$ time for node in 1 2 3 4 5 8 19 20 21 22 23 24 25 26 27 28 29 30 31; do ssh node$node hostname; done node1.internal node2.internal node3.internal node4.internal node5.internal node8.internal node19.internal node20.internal node21.internal node22.internal node23.internal node24.internal node25.internal node26.internal node27.internal node28.internal node29.internal node30.internal node31.internal real 0m8.004s user 0m3.830s sys 0m0.150s

As you can see the ssh version took just over 8 seconds, where the bpsh invocation took just under .1 seconds.

Now that I have sung the praises of bpsh, it does have one downfall. Java does not like it. There are a few solutions, my favorite being to just not use Java :). The other is to use ssh for anything that you need to do in Java.

In both MPI and Batch style jobs, there is a script that is written and submitted to the scheduler that does ALL of the work that the job needs to do. In comparison, in an Interactive job, the scheduler merely reserves the nodes and changes the permissions on each so that you can run commands on the nodes. Interactive jobs are usually used so that a person can become familiar with the computing environment and test their code before attempting a long production run.

When a script is submitted to the scheduler, the scheduler will take the request, and process it through it's algorithms and determine on which nodes it will run, and at what time the job will start. The same is done for an Interactive job as well.

The most basic Batch job will take a given set of nodes and run a command on each node. The following Batch job will use bpsh to run 'hostname' on each node.

#!/bin/bash
# Echo variables from scheduler
/bin/echo "Starting job:$JOBID on $NUMNODES nodes:$NODES"

# now run bpsh on each node
for node in $NODES;
do
	bpsh $node hostname
done
	

The MPI environment in Clubmask is lam-mpi. In order to write a script for a MPI job, the basics described above in the Batch job section will be used.

The difference between a basic Batch job and an MPI job lie in the script that is submitted to the scheduler. In order to use lma-mpi, there are a few specific setup steps, as well as a different method of running the program. Below is a sample MPI script that will run the classic MPI sample program cpi. This program can be found in the samples section. Please pay carefull attention to the comments in the script for a description of what each step is doing.

Example

#!/bin/bash # prog name PROG="/home/henken/pi" # Your jobid from the scheduler /bin/echo "JOBID:$JOBID" # The nodes the scheduler has assigned your job /bin/echo "NODES:$NODES" # create lamhosts file # This tells the lam environment what nodes you can use. # The '-1' is neccessary to tell lam that we are going to start the mpi # processes from the master node. echo "-1" >> /tmp/lamnodes.$JOBID for i in $NODES; do echo "$i">> /tmp/lamnodes.$JOBID done # boot up lam -- absoluteley necessary # This will start a lamd on each node. lamboot -b -x -v /tmp/lamnodes.$JOBID # run the mpi program with some args # see mpirun --help for more details # The recommended way of running MPI jobs utilizes one of the following two # mpirun commands. Notice the use of only nodes 1-X and cpus 1-x. This tells # lam to NOT execute a copy of the program on the master node. # run one copy per cpu mpirun c1-$NUMNODES $PROG # run on copy per node mpirun n1-$NUMNODES $PROG # clean up -- not absoluteley necessary, but a darn good idea lamclean -v #end lam session -- absoluteley necessary # if not done, running another mpi job may fail lamhalt -v

[henken@beta alpha]$ mpicc -o cpi ~/cpi.c 
[henken@beta alpha]$ cmsubmit -p 4 -t 10 cpi.sh              
_______________________________________________________________

Welcome to Clubmask!!!
_______________________________________________________________

==================================================
Job Summary
==================================================
Job Identification Number:  0212153059.262914
Username:  henken
Group:  2940
Job Classification:  Batch
Number of Processors:  4
Maximum Wall-clock Run-Time (minutes):  10
Program:  /home/henken/alpha/cpi.sh
Program Arguments:
Initial Working Directory: /home/henken/alpha
Job Output Directory: /home/henken
==================================================
Commit or Abort: ('c', 'a') ? c
please wait while database is updated...
please wait while database connection is closed...
		

[henken@beta alpha]$ more ~/0212153059.262914.stdout 
Starting job:0212153059.262914 with command:/usr/local/clubmask/packages/maui/run/0212153059.262914.program and args:./cpi to run on
nodes:11 11 16 16
JOBID:0212153059.262914
NODES:11 11 16 16
NUMNODES: 4

LAM 6.6b2cvs/MPI 2 C++/ROMIO/bproc - Indiana University

Executing hboot on n0 (-1 - 1 CPU)...
Executing hboot on n1 (11 - 2 CPUs)...
Executing hboot on n2 (16 - 2 CPUs)...
topology done      
Process 2 on 16
Process 0 on 11
Process 1 on 11
Process 3 on 16
pi is approximately 3.1416009869231249, Error is 0.0000083333333318
wall clock time = 2.386186
killing processes, done      
closing files, done      
sweeping traces, done      
cleaning up registered objects, done      
sweeping messages, done      

LAM 6.6b2cvs/MPI 2 C++/ROMIO/bproc - Indiana University

Shutting down LAM
LAM halted
		

The baic premise behind an Interactive job is asking the scheduler to reserve some set of nodes so that commands may be issued to the nodes from the command line. The first step is the request the nodes from the scheduler, as the following does for 4 processors for 30 minutes.

Example

[henken@beta henken]$ cmsubmit -p 4 -I -t 30 _______________________________________________________________ Welcome to Clubmask!!! _______________________________________________________________ ================================================== Job Summary ================================================== Job Identification Number: 0212155550.389526 Username: henken Group: 2940 Job Classification: Interactive Number of Processors: 4 Maximum Wall-clock Run-Time (minutes): 30 Program: Program Arguments: Initial Working Directory: /home/henken Job Output Directory: /home/henken ================================================== Commit or Abort: ('c', 'a') ? c please wait while database is updated... please wait while database connection is closed..

Now that the nodes are requested, you must wait until the job is started. The use of the showq and checkjob commands will give you a good idea when it will start.

Example

ACTIVE JOBS-------------------- JOBNAME USERNAME STATE PROC REMAINING STARTTIME 0212155550.389526 henken Running 4 0:30:00 Wed Feb 12 15:55:54 15 Active Jobs 4 of 40 Processors Active (10.00%) 2 of 20 Nodes Active (10.00%) IDLE JOBS---------------------- JOBNAME USERNAME STATE PROC WCLIMIT QUEUETIME 0 Idle Jobs BLOCKED JOBS---------------- JOBNAME USERNAME STATE PROC WCLIMIT QUEUETIME

Now that the job has reserved some nodes for us, we need to find what nodes we own. The cmgetnodes function takes a JOBID as an arguments, and returns the nodes for that job

Example

[henken@beta henken]$ cmgetnodes 0212160055.851849 11 11 16 16

Now that we 'own' nodes 11 and 16, it is time to do some work on them.

Example

[henken@beta henken]$ bpsh 11,16 hostname node11.internal node16.internal

With that exhausting amount of computation done, we will release the nodes back into the queue.

Example

[henken@beta henken]$ canceljob 0212160055.851849 job '0212160055.851849' cancelled

Note: The use of canceljob is not entirely necessary here, as the job will be stopped at the end of the allotted time, but it is a courtesy to the other users to return them when done. If your cluster administrator notices abuse of this 'nicety', the scheduler may be instructed to limit the time and number of Interactive jobs to prevent users from tying up the entire system and preventing the execution of scripted jobs.

cmsubmit is the program that is used to submit a job to the scheduler. The job will be added to the queue(s), and run when the scheduler determines the job is fit for execution.

#!/bin/bash
# CM -p 32
# CM -y
# CM -t 123

# Echo variables from scheduler
/bin/echo "Starting job:$JOBID on $NUMNODES nodes:$NODES"

# now run bpsh on each node
for node in $NODES;
do
	bpsh $node hostname
done
		

Once you have submitted your job, the next command you will use is 'showq'. Showq will show you the running and idle queues in the scheduler.

Note: Your job may take up to one minute to appear in the queue after submission.

Here is an example output from showq, the ACTIVE jobs are jobs that are currently running, IDLE jobs are those waiting for free nodes on which to run, and BLOCKED jobs are those jobs failing to run.

Example

ACTIVE JOBS-------------------- JOBNAME USERNAME STATE PROC REMAINING STARTTIME 1021213635.183226 feisha Running 2 1:51:50 Mon Oct 21 21:36:46 1021192537.570371 libin Running 2 2:07:00:54 Mon Oct 21 19:25:50 2 Active Jobs 4 of 46 Processors Active (8.70%) 2 of 23 Nodes Active (8.70%) IDLE JOBS---------------------- JOBNAME USERNAME STATE PROC WCLIMIT QUEUETIME 0 Idle Jobs BLOCKED JOBS---------------- JOBNAME USERNAME STATE PROC WCLIMIT QUEUETIME Total Jobs: 2 Active Jobs: 2 Idle Jobs: 0 Blocked Jobs: 0

For more information on showq, see showq

To cancel a job, you will use the 'canceljob' canceljob. Canceljob takes the jobid of a job as it's argument.

Example

[henken@beta henken]$ canceljob 0212155550.389526 job '0212155550.389526' cancelled

There are a few more scheduler commands that you will find usefull. The links for each command are to the official Maui scheduler webpages.