big_ingest.py - Helpers for handing a deposit to the big ingest

Description¶

Supports the hand-over from the regular to the big ingest pipeline.

Usage¶

python3 tools/big_ingest.py record-size <project-dir>
python3 tools/big_ingest.py needs-4x
python3 tools/big_ingest.py set-vm-size bitbucket-pipelines.yml <pipeline> 4x
python3 tools/big_ingest.py pipeline-variables

• record-size writes generated/deposit-size.csv with the columns project, zip_bytes, unzipped_bytes and small_bytes. small_bytes is the size of the files that are not larger than limitsize (see config.yml); git keeps a copy of those, larger files are excluded from it.

• needs-4x exits 0 if the deposit does not fit on a 2x worker, 1 if it does, and 2 on a configuration error. A missing size file counts as “does not fit”. The space needed is the larger of two peaks, plus a margin:

  • while unpacking: ZIP + unzipped files;

  • while committing: unzipped files + the git copy of the small files.

  It is compared with bigingest_threshold_mb (config.yml), with bigingest_margin_mb as the margin, or --threshold-mb / --margin-mb.

• The threshold is the free space of the build volume of a 1x/2x step, measured on a Bitbucket worker after the clone: 63,452,053,504 bytes (60,511 MiB) of a 67,049,664,512-byte volume. 1x and 2x are identical; 4x has 255,521,939,456 bytes free. The operating system, the image and the installed tools live on a separate volume and do not reduce this. The margin (2048 MB) covers the clone, logs, manifests and git’s temporary files.

• Whether the ZIP fits into the cache, which decides between the regular and the big pipeline, is detected by the pipeline itself and does not use this threshold.

• set-vm-size edits only the size: line of the named pipeline’s first step.

• pipeline-variables prints the JSON passed to the trigger pipe (jiraticket, openICPSRID, ... from the environment or config.yml).