Institute of Cancer Research (Alma HPC) Configuration
Deployment and testing of nf-core pipelines on the Alma cluster is on-going in collaboration between the RSE team and research groups.
To run an nf-core pipeline on Alma, run the pipeline with -profile icr_alma. This will download and launch the icr_alma.config which has been pre-configured with a setup suitable for the Alma HPC cluster.
Before running the pipeline you will need to load Nextflow using the environment module system. The preferred way is to create a condo or mamba environment (you only need to do this once) which you activate.
## Create and activate mamba environment
mamba create --name mamba_nf -c bioconda nextflow nf-core
mamba activate mamba_nf
Singularity is installed on the compute nodes of Alma, but not the login nodes. There is no module for Singularity.
All of the intermediate files required to run the pipeline will be stored in the work/ directory. Using the icr_alma profile, the Work directory will be deleted upon the successful completion of the run. If a run fails, this automatic cleanup will not execute - you must manually delete the work/ directory once debugged.
NB: Nextflow will need to submit the jobs via SLURM to the HPC cluster. This can be done from an interactive or a normal job. If in doubt contact Scientific Computing.
Alma has a master-worker partition for the long running nextflow manager which spawns jobs to the compute node. A typical nextflow job will run a batch script thorugh sbatch which will run on the master-worker thread spawning the jobs.
Command to run the nextflow job:
sbatch my-nextflow.sh
Contents of my-nextflow.sh
#!/bin/bash
#SBATCH --job-name=nf-username
#SBATCH --output=slurm_out.txt
#SBATCH --partition=master-worker
#SBATCH --ntasks=1
#SBATCH --time=1:00:00
#SBATCH --mem-per-cpu=4000
nextflow run nf-core/sarek -profile icr_alma,test --outdir ./results
Config file
/*
* -------------------------------------------------
* Nextflow nf-core config file for ICR alma HPC
* -------------------------------------------------
* Defines slurm process executor and singularity
* settings.
*
*/
params {
config_profile_description = "Nextflow nf-core profile for ICR alma HPC"
config_profile_contact = "Rachel Alcraft (@rachelicr), Mira Sarkis (@msarkis-icr)"
max_memory = 256.GB
max_cpus = 30
max_time = 5.d
}
process {
queue="compute"
executor = "slurm"
maxRetries = 3
maxErrors = '-1'
clusterOptions = '--mem-per-cpu=8000'
resourceLimits = [
memory: 256.GB,
cpus: 30,
time: 5.d
]
}
// Preform work directory cleanup after a successful run
cleanup = true
executor {
// This is set because of an issue with too many
// singularity containers launching at once, they
// cause an singularity error with exit code 255.
submitRateLimit = "2 sec"
perCpuMemAllocation = true
}
singularity {
enabled = true
runOptions = "--bind /mnt:/mnt --bind /data:/data"
autoMounts = true
pullTimeout = 2.h
cacheDir = '/data/scratch/shared/SINGULARITY-DOWNLOAD/nextflow/.singularity'
}
executor
contact
