Skip to article frontmatterSkip to article content
Site not loading correctly?

This may be due to an incorrect BASE_URL configuration. See the MyST Documentation for reference.

34_run_pii_checker.sh - Check data files for PII with a local LLM

Description

Checks the tabular data files of the deposit for personally identifiable information (PII) with pii-checker, which asks an LLM about every column that might hold PII. The LLM runs in Ollama on the machine that hosts the self-hosted Bitbucket runner, so the data never go to an outside service. The p-pii-checker pipeline runs it on that runner; Self-hosted runner for the PII check explains how to set one up.

The script:

  1. Downloads the deposit into a scratch directory outside the repository, with the downloader for the repository named in config.yml (openICPSR, World Bank, Dataverse or Zenodo). The checked-out deposit directory is not used: it holds only the code the ingest committed, possibly edited by the replicator, and no data. pii-checker unpacks archives in the deposit itself.

  2. Clones pii-checker (branch or tag PiiCheckerRef) into a scratch directory and installs its requirements into a virtual environment of its own.

  3. Runs tools/run_pii_checker.py. It loads the model in Ollama, checks that the model runs fast enough (see below), runs pii-checker with --strip-details, and sorts the results.

FileContentsWhere it goes
pii-checker-full/pii_checker_results.xlsxAll results, including the values from the data that were shown to the modelArtifact of the pipeline run, downloadable from the run’s Artifacts tab for 14 days (the Jira comment links to the run and gives the date). Never committed (.gitignore excludes pii-checker-full/)
generated/pii_checker_results-nodetails.xlsxThe same without the observed valuesCommitted by the pipeline
generated/pii_checker_summary.txtCounts of checked and flagged columnsCommitted by the pipeline, and posted to the Jira ticket

The reasoning column, which holds the model’s one-sentence explanation, stays in the committed workbook. It normally describes a column in general terms (“contains full names”), but review flagged columns before sharing the workbook further.

Ollama is reached at OLLAMA_ENDPOINTS if that is set. Otherwise the script uses port 11434 of the step container’s default gateway, which is the runner’s host. Ollama on the host must therefore listen on more than 127.0.0.1. The script loads the model before pii-checker starts: pii-checker’s own test call gives up after 30 seconds, and a large model can take longer to load from disk. When pii-checker finishes, it unloads the model.

Ollama decides once, when it loads a model, how much of it goes on the GPU, from the GPU memory that is free at that moment. If another model is loaded (for example from Open WebUI), most of the model can end up on the CPU: the check still works, but takes hours instead of minutes. So after loading, the script measures how many tokens per second the model generates for a short fixed prompt. Below PiiMinTokensPerSec, it unloads the model, waits PiiRetryMinutes, loads it again (so Ollama places it afresh) and measures again. If the model is still too slow after PiiWaitMinutes, the script exits with status 3 without running pii-checker, and writes why to generated/pii_checker_summary.txt, which the pipeline posts with its “aborted” comment. The message names the other models Ollama has loaded. The speed is measured, rather than the GPU memory, because measuring works with any GPU (or none) and Ollama does not report reliably how much of a model is on the GPU.

Usage

. ./tools/resolve_project_id.sh
./automations/34_run_pii_checker.sh $projectID

Environment Variables

tools/run_pii_checker.py

Can also be run on its own against a pii-checker checkout, with a Python that has pii-checker’s requirements:

python3 tools/run_pii_checker.py 123456 --checker-dir ../pii-checker --model gemma4:26b \
    --endpoint http://localhost:11434 --summary-file generated/pii_checker_summary.txt

Options: --full-dir (default pii-checker-full) and --output-dir (default generated) set where the two workbooks go. --min-tokens-per-sec, --wait-minutes and --retry-minutes correspond to the three variables above.