download_openicpsr-private.py - Download files from private openICPSR deposits

Description¶

This script authenticates with openICPSR and downloads all files from a private deposit as a ZIP archive, then extracts it to a local directory. It’s designed for downloading draft/unpublished deposits that require authentication and are not publicly accessible.

Usage¶

python3 tools/download_openicpsr-private.py PROJECT_ID [path] [login]

Arguments¶

PROJECT_ID (Optional) - Numeric openICPSR project ID (must be digits only) (default: derived from local config.yml)
path (Optional) - Download directory (default: current directory)
login (Optional) - Email for interactive authentication (will prompt for password)

Setup¶

You should normally have set up environment variables, see Setting up your Bash environment.

# Using environment variables for authentication
export ICPSR_EMAIL="your.email@domain.com"
export ICPSR_PASS="your_password"
export ICPSR_TOKEN="your_token"

Project-specific setup¶

Usually, the config File config.yml will have openicpsr: PROJECT_ID already set. If not, this can be edited and committed to the repository. The script will read that config file to determine the PROJECT_ID.

Examples¶

# Default usage, reads openICPSR deposit number from config.yml
python3 tools/download_openicpsr-private.py 

# Overriding the default PROJECT_ID from config.yml, or empty config.yml
python3 tools/download_openicpsr-private.py 123456

# Specifying download path
python3 tools/download_openicpsr-private.py 123456 ./downloads

# Interactive login (will prompt for password)
python3 tools/download_openicpsr-private.py 123456 ./downloads your.email@domain.com

Authentication Methods¶

The script supports multiple authentication methods in order of preference:

Interactive Authentication: Command line login + password prompt
Environment Variables: ICPSR_EMAIL and ICPSR_PASS

Environment Variables¶

ICPSR_EMAIL - Your openICPSR account email
ICPSR_PASS - Your openICPSR account password
ICPSR_TOKEN - Optional Cloudflare bypass token (see ICPSR_TOKEN below)
DEBUG - Enable debug output (any non-empty value)

ICPSR_TOKEN¶

openICPSR sits behind Cloudflare, which may block automated requests (for example, from cloud-hosted pipeline runners) before they reach the login page. ICPSR_TOKEN carries a token, agreed with openICPSR, that lets the script’s requests bypass that check.

Strongly suggested: if ICPSR_TOKEN is unset or empty, the script runs as before and sends no extra header. If downloads fail with a Cloudflare challenge or HTTP 403 response, set the token.

Config File Format¶

openicpsr: 123456  # Your project ID

Requirements¶

Python 3.x
requests: HTTP client library
PyYAML: YAML configuration file support
Valid openICPSR account with access to target deposit

This tool is essential for research workflows that require access to unpublished or private openICPSR deposits during the replication process.