Skip to article frontmatterSkip to article content
Site not loading correctly?

This may be due to an incorrect BASE_URL configuration. See the MyST Documentation for reference.

compare_manifests.py - Report what changed between two manifests

Description

Compares two SHA256 manifests (sha256sum format, as written by automations/02_create_manifest.sh) and reports what changed in a meaningful way. Usually this is the previous and current ingest of a deposit, for example across revision rounds.

Every file in the two manifests ends up in exactly one category:

CategoryRuleMeaningful?
Unchangedsame path, same checksumno (count only)
Content changedsame path, different checksumyes
Renamedsame checksum, different file name (possibly also a new directory)yes
Moved and changedpath gone; a file with the same name appears elsewhere with different content, and the name is unique on both sidesyes
Added / Removedeverything elseyes
Moved onlysame checksum and file name, different directoryno (listed grouped by directory)

Matching is done on full paths first. Files are paired by checksum only when their path changed, so files that share a name in different folders (for example firms.txt in each yearly folder) are never cross-matched.

Added files whose content is identical to a file in the previous manifest are flagged as copies. Added or removed files that make up a wholly new or wholly removed directory are listed as that directory with a file count, e.g. “input/adoinstalled/ (new directory, 292 files)”.

Usage

python3 tools/compare_manifests.py PREVIOUS CURRENT [--summary-file FILE]

The full report (Markdown) goes to standard output. automations/22_compare_manifests.sh writes it to generated/manifest-comparison.md, which is included in the report appendix, and writes the summary to generated/manifest-cmp-summary.md.

Example output

| Category | Files |
|---|---:|
| Files in previous manifest | 742 |
| Files in current manifest | 742 |
| Unchanged (same path and content) | 740 |
| **Content changed** (same path) | 2 |
| **Renamed** (same content, new name) | 0 |
| **Moved and changed** (same name, new location, new content) | 0 |
| **Added** | 0 |
| **Removed** | 0 |
| Moved only (same name and content, new location) | 0 |

**Content changed** (2):

- `README.pdf`
- `_step_1_pt_16_descriptives.do`