BAYWULF FAQ

You must be on a lab machine to have ssh access to the cluster. If you are coming in from an outside machine, send an email to access if you would like your machine to have ssh access to the cluster.

Please send emails to help! for questions about the cluster. Response times may vary between 1 minute and 1 week.

Since the cluster is running NIS with the user information stored on master, you must use yppasswd to change your password there. You can run yppasswd on any of the nodes in the cluster.

When you are given an account, you receive 5 Gb of disk space on master2. To check your current quota, use the quota command. This will show you how much space you have left on master2. quota works on all the nodes, except right now there is a bug in the remote quota daemon that report 1/4 the amount of space that is actually there.

There is a bug in the rpc.rquotad running on master2 that reports your disk usage in 4 kb rather than 1 kb blocks. Therefore, when using the quota command on any of the nodes other than master2, your quota reporting is actually 1/4 that of what it actually is.

All data on master2 (your home directories) is backed up weekly, with incremental backups performed on a daily basis. Data cannot be stored on the nodes, as all data that is more than a week old is deleted from the scratch directories. While your scratch directories will not be deleted, the files within them will. Do not rely on touch scripts that periodically touch your scratch data, as these scripts will be killed indiscriminantly.

No. All sleep scripts will be deleted indiscriminantly.

The only machine that can see the outside world is master. You must be working on master to be able to set the display to your remote machine or to ssh to your remote machine.

Once you ssh to baywulf, you will be on master. You can then rsh to any of the other machines without using your password. Your home directory, however, will always be on master2:/usr/home/users/your_home.

The cluster is composed of a total of 20 space sharing cluster nodes. This means that jobs are assigned to the processors according to the amount of space they need and the space that is currently available. The opposite is time-sharing nodes, in which jobs are allocated to the nodes and load balanced according to a specified load on each machine.

The batch queueing system is PBS. There is plenty of information on the web concerning PBS, so if you can't find your answers here you can go to the PBS documentation site.

The first thing you need to do is write a script to submit your job to the system. Here is a resubmission script to get you started. Files required for this script to run are: run.txt (set to 1 to start), maxruns.txt, containing the maximum number of jobs you would like to submit to the queue).

#!/bin/sh

To submit your job, from master, type:

qsub myjob.script

The appropriate queue will be chosen according to the space you need. If you need less than 1024 mb, your job will be sent to psque, otherwise, your job will be sent to sque. Please do not use sque unless you have special permission.

You can find out about the current queues by typing:

qstat -q

To find out the status of your job, use:

qstat -a, or just qstat

To find out detailed information about your job, use:

qstat -f

The above information can all be shown with the xpbs command, a graphical interface to monitor the pbs queue.

Before you submit your job, you need to decide which node you would like to run on. To determine the least loaded node, use the xpbsmon GUI to see a picture of the nodes and their loads. Or, alternatively, you can use the command:

pbsnodes -a

This will give you a list of all the nodes and their availability. If you submit a job to a node that is currently loaded, your job will wait in the queue until the jobs ahead of it are completed. Once you have determined which node you would like to run on (currently the queue does not automatically do this for you), specify it in your script with the line:

#PBS -l nodes=node15

Or, alternatively, for parallel jobs, you can specify the number of nodes you want with

#PBS -l nodes=4

But this will place your job on the first four processors regardless of their load. To specify which nodes you want, use

#PBS -l nodes=node11+node12+node13+node14

You can do this on the command line with the qsub command with:

qsub -l nodes=node11+node12+node13+node14 myjob.script

You can also specify the number of CPUs you want to use per node with

#PBS -l nodes=node11:ppn=2+node12:ppn=2

This will run your job on four processors, two on node11 and two on node12. Since currently there is no way to designate nodes to specific queues, it is possible to submit your jobs to all of the nodes in the cluster. Please limit your jobs to the nodes available in the xpbsmon command excluding master1,master2,node3, and node4.

Each node has 1 Gb of total RAM. You are free to use all of the RAM on each machine. However, there is a bug with Linux memory management that does not free unused memory. Thefore, if you specify the amount of memory your job needs in your PBS script with

#PBS -l mem=200mb

your job may be placed in the queue even though no other jobs are running on that machine. The solution, albeit an unelegant one, is to say you only need 10mb in your script even though your job needs more. PBS will run your job even if it is bigger than 10 mb.

Use the size command on your executable. For example,

size a.out

results in

text data bss dec hex filename

which shows that your executable uses 19952660 bytes of statically allocated RAM. If you are using F90 or some other compiler which allows dynamic allocation, you need to determine how much space you are going to need either by looking at your code or by checking how much your job uses while it is running with the top command.

You should use the Compaq compiler fort to compile both your fortran 77 and fortran 90 programs. C and C++ programs can be compiled with gcc or g++, respecively. To compile and link your MPI programs to the MPI libraries, use mpif77, mpif90, or mpicc. The mpi scripts use fort to compile and link your fortran MPI programs, and gcc or g++ to compile and link your MPI C or C++ programs.

You can compile your codes on any machine, and you do not need to submit a job to compile your codes. However, it is adviseable to compile your codes on master since it has the most updated software and libraries. If you compile your code on master and it does not run on one of the nodes, please inform admin of the problem.

Because all the nodes and masters have the home directories mounted from master2, it is not efficient to write to your home directories on the masters and the nodes when running your jobs, as this will clog the cluster network. Therefore, you should set up your jobs so that they write to your directories in the local scratch partitions. That is, if my userid is babywulf, then I need to set up my jobs so that they all write to /scratch/babywulf. The /scratch directories exist on all of the machines, except they point to different partitions depending on which machine you are running on, as shown below:

• master1
  • /scratch mounted on /dev/sda5, 28 Gb SCSI
  • links: /usr/scratch -> /scratch
• master2
  • /scratch does not have its own partition. You are writing to your home directories when you write to /scratch on master2.
  • links:
    /usr/scratch -> /usr/home/users
    /scratch -> /usr/home/users
• nodes2-20
  • /scratch mounted on /dev/hda5 13 Gb IDE
  • links: /usr/scratch -> /scratch

You can check the status of your executables on the processors either by using rsh from any machine in the cluster or bWatch.tcl from master. To check the status of your a.out job, say on node3, from any node you can use

rsh node3 ps ax | grep node3

To check the load on all the machines in the cluster, you must be on master, and your display must be set to your remote machine. To use bWatch.tcl, type

setenv DISPLAY $REMOTEHOST:0.0

This will bring up a window on your machine displaying the load and memory status for each machine in the file /usr/rels/mpich/util/machines/machines.LINUX_ALPHA.

You can run a command on a remote machine without actually logging into that machine with rsh. For example, to see if your a.out process is running on node12, you can type

rsh node12 ps ax | grep a.out

This will execute the given command on that machine and display the result on the machine you're on. You can run a command on all the machines in the cluster with the psync script located on master. For example, to see if your a.out process is running on all the nodes, just type

psync "ps ax | grep a.out"

This will run the command on all the machines listed in the file /mpi/util/machines/machines.LINUX_ALPHA.

Alternatively, you can run a command on the nodes you specify by using the sync.sh script on master. This works by specifying the nodes in a separate file and using the same syntax as psync. To run the above command only on nodes3 and 4, type

sync.sh "ps ax | grep a.out" hosts.txt

Here, hosts.txt is a file containing the two nodes that you want to run the command on. It's syntax is as follows:

node3

Remember that running remote commands with rsh will not work with interactive output such as that with top or more. To view a file on another machine, you can use

rsh node5 "cat filename"

If we attempted to view a file with more on another machine with

rsh node5 "more filename"

it would hang because the other machine is waiting for a response from the standard input to proceed.

To copy a file from one machine to another, you can use the rcp command. For example, if you are on node5 and you want to copy the file input.dat from node3 to the local scratch directory, use

rcp node3:/usr/scratch/username/input.dat /usr/scratch/username/.

If you want to copy a file from one machine to another, use

rcp node3:/usr/scratch/username/input.dat node5:/usr/scratch/username/.

To copy a file from the baywulf cluster to a machine outside the cluster, you must use scp to or from master. To copy a file hello.txt from the cluster to your machine, you must be on master, and type

scp hello.txt mymachine:.

To copy a file from your machine to the baywulf cluster, type

scp hello.txt baywulf:.

and the file will be left on your home directory on the cluster. To copy multiple files or directories, use the same as above, only use scp -r for recursive copy.