sbatch-shell.sh - SLURM job template with Jira notifications

Description¶

tools/sbatch-shell.sh is the template for submitting a replication run to a SLURM cluster (e.g. BioHPC). It carries the usual #SBATCH resource directives.

Usage¶

A Stata example:

cd /path/to/aearep-1234
cp tools/sbatch-shell.sh run-stata.sh
# edit --mem/--cpus-per-task/--time, uncomment the Stata line, comment out Matlab/R
sbatch run-stata.sh

The job will send emails to the submitter, and will post two comments to the ticket’s Part B sub-task.

What to edit¶

See ECCO SBATCH examples for more details on how to configure these values:

JIRA-related parameters, can usually be left as-is:

Jira notifications¶

Two calls, both to jira_add_comment.py, wrapped in a jira_notify shell function so that the common options appear once:

• Start: jira_notify --status started, right before the payload.

• Stop: jira_notify --status completed --exit-code "$rc", from a trap.

The stop notification runs from a trap on EXIT and TERM so it also fires when the job fails or is killed at the wall clock limit (reported as exit code 143). --exit-code is what turns “completed” into “failed”, so a single call covers both outcomes. The final exit $? in the template makes the last command’s status the job’s status.

Both calls pass --partb, which redirects the comment from the main ticket to its Part B sub-task, and --slurm, which folds the job ID, job name and submit directory into the status line itself, e.g. ✅ SLURM job 590340 main.do completed (directory: /path/to/submit/dir).

Ticket¶

With JIRATICKET=auto, the ticket is taken from $jiraticket if set, otherwise from the jiraticket: line of the nearest config.yml (searched from $SLURM_SUBMIT_DIR upward). Submitting from the replication repository is therefore enough.

Credentials¶

JIRA_USERNAME and JIRA_API_KEY are read by jira_add_comment.py from the environment, from ./.env, or from ~/.envvars - nothing needs to be set in the SLURM script. On a cluster, ~/.envvars is the usual place, because compute nodes do not inherit an interactive login shell.

If credentials cannot be found, the job runs normally and the notifications are skipped with a warning in the SLURM log.

Graceful degradation¶

jira_notify never blocks, retries, or raises: if credentials are missing, jira_add_comment.py prints a warning and returns immediately, and jira_notify itself ends in || true so even a Python-level crash cannot propagate. jira_notify_end captures the payload’s real exit code in rc before calling jira_notify, and does exit "$rc" after it - so the job’s own exit code is never touched by the Jira calls, whether or not they succeed.

Verified directly: running the template outside SLURM with no JIRA_USERNAME/JIRA_API_KEY reachable anywhere (env unset, HOME pointed at an empty directory so no credential file is found either) and a payload that exits 5:

Warning: Jira credentials not available, skipping comment
payload-ran
Warning: Jira credentials not available, skipping comment
$ echo $?
5

The payload still ran to completion, and the job’s exit code was exactly what the payload returned.

Python on the compute node¶

sbatch-shell.sh calls jira_add_comment.py with the compute node’s native python3 directly - no module load is needed. The tool only needs the standard library (it falls back to the Jira REST API when the jira package is not importable), and runs on Python 3.6+, so an older native interpreter is fine.

Confirmed end-to-end on BioHPC/ECCO’s native Python 3.9.25, including with the jira package (which happens to be importable there without any setup). A module load step was tried during development and dropped again: it only added a cluster-specific MODULEPATH dependency (/programs/modulefiles on BioHPC/ECCO is not on the default path) for no benefit, since native Python 3.9 already does everything this script needs. If your cluster’s native python3 is older than 3.6, point PYTHON_CMD in the template at a compatible interpreter yourself.

See Also¶

• jira_add_comment.py - the tool that does all of the Jira work

• SLURM quick start