29_prepare_big_ingest.sh - Hand a too-big deposit over to the big ingest

Description¶

Run by the “Check downloads” step of the regular ingest pipeline when the deposit ZIP did not fit into the cache. It

1. checks whether the deposit, from the recorded sizes, needs more disk than a 2x worker has free (bigingest_threshold_mb and bigingest_margin_mb in config.yml, see big_ingest.py - Helpers for handing a deposit to the big ingest); if so it sets the big pipeline’s VM to 4x in bitbucket-pipelines.yml, commits (through 20_commit_code.sh) and pushes;

2. posts a Jira comment (through 70_publish_comment.sh) that the regular pipeline aborted because the deposit is too big.

The pipeline then triggers the big ingest with the atlassian/trigger-pipeline pipe, whether or not the VM was changed, and fails the regular pipeline.

Usage¶

./automations/29_prepare_big_ingest.sh <regular-pipeline-name> <big-pipeline-name>

Requirements¶

The pipe triggers the big ingest as the shared system user (the AEA System User), not as whoever started the regular pipeline, so it works the same for every user. It uses two workspace variables:

• JIRA_USERNAME: the system user’s Atlassian account email (passed to the pipe as ATLASSIAN_ACCOUNT_EMAIL).

• P_BITBUCKET_PAT: an Atlassian API token for that account that is accepted by the Bitbucket API (passed as ATLASSIAN_API_TOKEN). The Jira API token (JIRA_API_KEY) is not accepted by Bitbucket, so it cannot be used here. The account must have write access to the repositories. The same variable is what aeagit-create uses locally.

The account must also be a workspace member that can see and run pipelines in every aearep-NNNN repository, for example through a user group with write access (such as developers). Repositories are private, so an account without access gets “Account, repository or branch doesn’t exist” (HTTP 404) from the pipe, even though its token is valid.

If the token is not accepted by Bitbucket, the pipe fails with an HTTP 401 error. In either case the regular pipeline still fails, but the big ingest is not started.