GPU Programming and CUDA related documentation

 With the new iceberg upgrade, we acquired 8 NVIDEO GPU units that can be used for HPC programming using the CUDA or PGI Fortran Compilers on iceberg. Currently only two of the new INTEL nodes hosts the GPU units.
Model number of our GPUs are Tesla M2070 which support Compute Capability at level 2.0 .

Getting access to the GPU units

In order to ensure that the node(s) hosting the GPUs run only GPU related tasks, we defined a special project-group for accessing these nodes. If you wish to take advantage of the GPU processors, please contact iceberg-admins@lists.shef.ac.uk  asking to join the GPU project group.

Qsub or Qsh parameters

Once you are included in the GPU project group you may start using the GPUs by typing;
 qsh  -l gpu=n   -l arch=intel*   -P gpu   to start an interactive job.

Or

qsub  -l gpu=n    -l arch=intel*   -P gpu  scriptfile    to start a batch job that runs the commands in the scriptfile that is provided, where n is the number of GPU units to be used by the job. Currently maximum GPU units allowed per job is set to 4
i.e. you can not exceed gpu=4. We will normally expect each gpu job to use a single GPU unit. So, use gpu=1 . 
     The first line of the scriptfile must contain the so-called bang line that defines the shell which will normally be #!/bin/bash   but may also be  #!/bin/csh   or #!/bin/tcsh  for running scripts that use the c-shell.

Having started the job, to setup the NVIDEO Cuda compiler environment you will need to use one of the two module statements given below;

module add libs/cuda/3.2.16
module add libs/cuda/4.0.17

The cuda compiler can be accessed by typing ;

 nvcc filename.cu -arch sm_20 
The -arch flag above signifies the compute capability of the intended GPU hardware, which is 2 for our current GPU modules. If you are intending to generate code for older architectures you may have to specify  sm_10 or sm_13 for example. 


Similarly the PGI compilers with GPU extensions are available via ;

module add compilers/pgi/10.2 

From time to time as we update the software & compilers we will add/remove modules.
To see which modules are currently available type module avail  .

Documentation on GPU and CUDA programming

We have not yet produced local documents on the use of the GPU units. However, the following collection of external links contain useful information. 
Documentation for GPU and CUDA programming