Load balancer

This is a simple example illustrating how to use the load balancer tool. Essentially you enumerate the commands you would like to execute in parallel in a file, making sure that you only put one command per line. If you have multiple commands as part of the same task, separate them with a semicolon (;). This means that you can use your favorite scripting language to create your set of tasks, or you can simply copy-and-paste if you only have a few.

There are two ways to execute this code: interactively and batched. More information about submitting jobs in the Research Computing environment is available in the documentation for batch queueing and job scheduling.

  1. Create the cmd_lines file in the directory you created.
    $ for  i in {1..100} ; do \
     echo "sleep 2; echo process $i" >> cmd_lines ; \
     done
    Your ending file should look like the following:
    sleep 2; echo process 1
    sleep 2; echo process 2
    sleep 2; echo process 3
    ...
    sleep 2; echo process 99
    sleep 2; echo process 100
    
  2. Load the load balancer module.
    $ module load loadbalance
  3. Submit your job, for example using srun.
    $ module load slurm
    $ srun -N 1 --ntasks-per-node=12 lb cmd_lines
  4. You may also submit your job by creating and submitting an sbatch script:
    $ module load slurm
    $ module load loadbalance
    $ vi submit.sh
    Your sbatch script should look something like this:
    #!/bin/sh
    #SBATCH -N 1
    #SBATCH --ntasks-per-node 12
    #SBATCH --output output.out
    #SBATCH --qos janus
    
    srun lb cmd_lines
    Finally submit your script.
    $ sbatch submit.sh