Description¶
Checks the tabular data files of the deposit for personally identifiable information (PII) with pii-checker, which asks an LLM about every column that might hold PII. The LLM runs in Ollama on the machine that hosts the self-hosted Bitbucket runner, so the data never go to an outside service. The p-pii-checker pipeline runs it on that runner; Self-hosted runner for the PII check explains how to set one up.
The script:
Downloads the deposit into a scratch directory outside the repository, with the downloader for the repository named in
config.yml(openICPSR, World Bank, Dataverse or Zenodo). The checked-out deposit directory is not used: it holds only the code the ingest committed, possibly edited by the replicator, and no data. pii-checker unpacks archives in the deposit itself.Clones pii-checker (branch or tag
PiiCheckerRef) into a scratch directory and installs its requirements into a virtual environment of its own.Runs
tools/run_pii_checker.py. It loads the model in Ollama, checks that the model runs fast enough (see below), runs pii-checker with--strip-details, and sorts the results.
| File | Contents | Where it goes |
|---|---|---|
pii-checker-full/pii_checker_results.xlsx | All results, including the values from the data that were shown to the model | Artifact of the pipeline run, downloadable from the run’s Artifacts tab for 14 days (the Jira comment links to the run and gives the date). Never committed (.gitignore excludes pii-checker-full/) |
generated/pii_checker_results-nodetails.xlsx | The same without the observed values | Committed by the pipeline |
generated/pii_checker_summary.txt | Counts of checked and flagged columns | Committed by the pipeline, and posted to the Jira ticket |
The reasoning column, which holds the model’s one-sentence explanation, stays in the committed workbook. It normally describes a column in general terms (“contains full names”), but review flagged columns before sharing the workbook further.
Ollama is reached at OLLAMA_ENDPOINTS if that is set. Otherwise the script uses port 11434 of the step container’s default gateway, which is the runner’s host. Ollama on the host must therefore listen on more than 127.0.0.1. The script loads the model before pii-checker starts: pii-checker’s own test call gives up after 30 seconds, and a large model can take longer to load from disk. When pii-checker finishes, it unloads the model.
Ollama decides once, when it loads a model, how much of it goes on the GPU, from the GPU memory that is free at that moment. If another model is loaded (for example from Open WebUI), most of the model can end up on the CPU: the check still works, but takes hours instead of minutes. So after loading, the script measures how many tokens per second the model generates for a short fixed prompt. Below PiiMinTokensPerSec, it unloads the model, waits PiiRetryMinutes, loads it again (so Ollama places it afresh) and measures again. If the model is still too slow after PiiWaitMinutes, the script exits with status 3 without running pii-checker, and writes why to generated/pii_checker_summary.txt, which the pipeline posts with its “aborted” comment. The message names the other models Ollama has loaded. The speed is measured, rather than the GPU memory, because measuring works with any GPU (or none) and Ollama does not report reliably how much of a model is on the GPU.
Usage¶
. ./tools/resolve_project_id.sh
./automations/34_run_pii_checker.sh $projectIDEnvironment Variables¶
PiiModel- Ollama model (default:gemma4:26b). It must already be pulled on the host.PiiCheckerRef- Branch or tag of pii-checker (default:main).PiiMinTokensPerSec- Slowest acceptable generation speed, in tokens per second;0skips the check (default:40;gemma4:26bmakes about 100 on two RTX 3060 GPUs). What is fast enough depends on the model and the machine: compare the speed the step logs (“... generates N tokens/s”) when the model is on the GPU and when it is not.PiiWaitMinutes- How long to keep reloading a model that is too slow (default:30).PiiRetryMinutes- Pause between reloads (default:5).OLLAMA_ENDPOINTS- Ollama URL, for examplehttp://192.168.1.4:11434(default: the default gateway, port 11434).openICPSRID,WorldBankID,DataverseID,ZenodoID- Set bytools/resolve_project_id.sh; the first one set picks the downloader.ICPSR_EMAIL,ICPSR_PASS- For a private openICPSR deposit.
tools/run_pii_checker.py¶
Can also be run on its own against a pii-checker checkout, with a Python that has pii-checker’s requirements:
python3 tools/run_pii_checker.py 123456 --checker-dir ../pii-checker --model gemma4:26b \
--endpoint http://localhost:11434 --summary-file generated/pii_checker_summary.txtOptions: --full-dir (default pii-checker-full) and --output-dir (default generated) set where the two workbooks go. --min-tokens-per-sec, --wait-minutes and --retry-minutes correspond to the three variables above.
Related¶
16_run_pii_scanner.sh runs the Stata PII scanner, which checks variable names and labels only, without an LLM.