Troubleshooting¶
Recommended steps to troubleshoot the pipeline.
Email¶
Check your email for an email regarding pipeline failure. You will receive an email from slurm@biowulf.nih.gov with the subject: Slurm Job_id=[#] Name=CARLISLE Failed, Run time [time], FAILED, ExitCode 1
Review the log files¶
Review the logs in two ways:
- Review the master slurm file: This file will be found in the
/path/to/results/dir/
and titledslurm-[jobid].out
. Reviewing this file will tell you what rule errored, and for any local SLURM jobs, provide error details - Review the individual rule log files: After reviewing the master slurm-file, review the specific rules that failed within the
/path/to/results/dir/logs/
. Each rule will include a.err
and.out
file, with the following formatting:{rulename}.{masterjobID}.{individualruleID}.{wildcards from the rule}.{out or err}
Restart the run¶
After addressing the issue, unlock the output directory, perform another dry-run and check the status of the pipeline, then resubmit to the cluster.
# unlock dir
carlisle --runmode=unlock --workdir=/path/to/output/dir
# perform dry-run
carlisle --runmode=dryrun --workdir=/path/to/output/dir
# submit to cluster
carlisle --runmode=run --workdir=/path/to/output/dir
Contact information¶
If after troubleshooting, the error cannot be resolved, or if a bug is found, please create an issue or send and email to CCBR_Pipeliner@mail.nih.gov
Last update: 2025-01-09