Preparing your files for verification - details

All datasets actively used in your replication package that are not your own must be cited. Just as with any other literature, this serves to properly identify the provenance of the information you use, and attribution of credit to the original creator. Data you created should also be cited, either by pointing to a data deposit you made elsewhere, or by pointing to your own (forthcoming) data deposit at the AEA. Data you collected should be cited with the original location where you obtained the data, even if you provide a copy in your data deposit.

• AEA Sample References

• Guidance for data citations

• Guidance on how to cite the data you collected

• Some additional considerations NO SUCH DIRECTORY

Every citation has an in-text entry (Smith (2020) or (Smith, 2020)), and a list entry in the Reference section (see the Chicago Manual of Style Quick Guide). This is true for data citations as well.

• Your data citations should be listed in the Reference section of the manuscript.

In rare circumstances, when there are too many data citations to accomodate succinctly, data citations for sources can be deferred to an online data appendix. In all cases, for clarity, all data citations should also appear in the README, including with a separate Reference section.

In many cases, the data citation for a public-use dataset has a unique URL that is sufficient for downloading the dataset:

Bureau of Labor Statistics. 2000–2010. “Current Employment Statistics: Colorado, Total Nonfarm, Seasonally adjusted - SMS08000000000000001.” United States Department of Labor. http://data.bls.gov/cgi-bin/surveymost?sm+08 (accessed February 9, 2011).

However, in many other cases where the URL does not lead directly to the dataset, a separate “Data Availability Statement” needs to be provided, as part of the README. A simple data citation may not sufficient. See general guidance on data availability statements, and some examples in the template README.

This includes

• code to read in raw data,

• code to clean and process data,

• code to analyze data and estimate models, and

• code to generate tables and figures.

This applies even to

• maps and other complex visualizations (guidance)

• web scraping code

• code that processes confidential data (see below)

• code that anonymizes data

• code that prepares surveys and runs experiments (guidance)

In all cases, it is possible that “code” is not a program, but human-readable instructions on how to perform a manipulation.

• For instance, despite the availability of scriptable GIS software, many economists prefer to use GUI-based GIS software, and manually construct maps.

• For instance, many websites providing data have only an interactive process to acquire or request data, and no single download link. In such cases, detailed instructions on how to acquire the data may be necessary (see Daniels et al. (2023) for an example). We do encourage authors to use APIs where available.

Code should be included in its “native” form, i.e., the format that the software expects to read it in. For instance, Stata code should be in .do files, R code in .R files, Qualtrics surveys in .qsf files, etc.

The replication package should reproduce the tables and figures, as well as any in-text numbers, by running code without manual intervention.

The only exception to this rule is a single change to set a small number of program and data directory paths.

While running a small number of distinct programs separately is acceptable (in some cases even desirable), it is not acceptable to require replicators to manually enter numbers, or configure parameter files, in order to reproduce the tables and figures in the paper. Nevertheless, it should be clear from the manuscript and the code how a replicator might deviate from the tables and figures in the paper.

ICPSR cannot accept deposits with more than 1,000 files. Therefore, we relax the rule that all data and code must be unzipped, though we still insist on the “smallest possible configuration”.

In most cases, it is a particular directory that is the primary culprit. Say you have

Structure-pre:

/code: 20 files
/data/
   src1/: 25 files
   src2/: 101 files
   src3/: 3,000 files

then the ideal structure, taking into account the 1,000 file limit, would be:

Structure-post:

/code: 20 files
/data/
   src1/: 25 files
   src2/: 101 files
   src3.zip: = 1 file, containing 3,000 files

(src3/ and its 3,000 files have been removed!)

Your README should provide instructions to the replicators how to recover the fully unzipped structure (there are cross-platform differences in unzipping, so be precise about the final structure, rather than the method of getting there).

Alternatively, the code can handle the unzipping - optional, but more robust.

Once you’ve adjusted that, zip up the whole structure (so a ZIP file that has inside it another zip file, plus the /code, /data/src1/, and /data/src2/ directories), and “Import from ZIP” when uploading to ICPSR.

• See also a similar entry to our FAQ.

When the replication package relies on confidential data that cannot be shared, or is shared under different conditions, authors will have to

• prepare a confidential (partial) replication package

  • this would contain the contents of data/confidential and possibly data/conf_analysis from the example below.

• preserve (archive) the confidential replication package

  • If the data cannot be removed from a secure enclave, they should nevertheless be archived wherever the confidential data are kept (see this FAQ)

  • If the data can be shared, but are subject to access restrictions, follow this guide on creating a separate data deposit and, when creating the restricted deposit at ICPSR, follow these instructions on how to do so.

• prepare a non-confidential replication package that contains all code, and any data that is not subject to publication controls

  • this would contain the contents of data/raw, data/analysis, code/, and for reference, results/ from the example below.

• ensure that replicators have detailed instructions on how to combine the two packages

• specify which (if any) of the results in their paper can be reproduced without the confidential data.

Clearly separate the restricted from the open-access data, both in terms of the raw data as well as the processed data:

data/
   raw/
      cps0001.dat
   confidential/
      ssa.csv
   conf_analysis/
      confidential_combined.dta

Keep in mind that you may be able to provide a subset of your replication package privately to the AEA Data Editor, see the Sharing restricted-access data with the AEA Data Editor page, or that you might be able to create a separate data deposit with a more limited license.

Authors might want to investigate the possibility of providing “fake” or “simulated” data that might allow replicators to run code, without necessarily obtaining meaningful results (functionality test).

• Sharing restricted-access data with the AEA Data Editor

• Licensing guidance.

• The Social Science Editors’ FAQ describes a related issue.

In situations where the data has a different license, or where the data you include has value separately from your replication package, you can create a separate data deposit, and link it/reference it/cite it.

• Creating a separate data deposit

• Guidelines for depositing at other repositories

• Special case: PSID

• Special case: confidential data

• that the replicator has basic knowledge in how to run your software package, if the software is commonly used in economics

  • Stata, Matlab, some others are commonly used

  • Compiled or new computer languages are much less likely to be widely used, even if they are used in your subdiscipline

• that the replicator can manipulate a top-level configuration file

  • for instance, to set a base directory

  • but not setting a base directory at the top of 25 different files

• that the replicator will use the same type of operating system

  • describe any hard requirements, but do not impose any fake requirements

  • most Stata, Matlab, SAS, R, Python, etc. can run on any operating system, unless you hard-code platform-specific commands into your code

• that the replicator has any of your packages/modules/etc. installed

  • provide a setup program to install these (not manual instructions).

  • provide copies of such packages/modules when the package repository does not allow you to specify a version

  • provide a container/ Docker image/ VM that comprises all the necessary software and libraries

• that the replicator will run the software the same way you do

  • some software can be run in different ways (interactive, batch, etc.) and may behave differently depending on how it s run.

  • For instance, Stata will change the working directory to that of the program being run in batch mode, but not if running interactively

  • For instance, running R code using Rstudio may behave differently than running it with rscript