Protocol for Third-party Verifications
This protocol describes how third parties can, at the request of the AEA Data Editor, conduct a reproducibility check of materials that are part of an AEA publication.
Alternate protocols are possible, but should be verified with the AEA Data Editor prior to engaging any resources.
- The author of the replication archive or archives should provide a complete and exhaustive archive, ready for publication.
- The archive does not need to be public at this stage, as long as it can be shared privately.
- The AEA Data Editor will have verified the archive.
What or Who Is a Third-party Replicator
The third-party replicator is somebody not affiliated with the AEA editorial offices (third-party), who has access to the (confidential or restricted) data used in the author’s paper, and who
- is not involved with the author’s research project
- has disclosed any conflicts of interest
- promises to conduct an arms-length reproducibility exercise, with minimal to no interaction with the author (other than to undertake specific steps to access the data)
Steps for the Third-party Replicator
- Download the author’s replication archive(s) from the designated URL (public, or privately shared)
- Ensure access to any confidential files that are described in the replication archive’s README
- The replicator should consider whether a third-party person not familiar with the precise data environment could reasonably find and access the data, based solely on the instructions in the README
- Follow the checklist to conduct the reproducibility exercise, relying exclusively on the README for instructions and guidance.
- Write a report
- Send the report to the AEA Data Editor
- Report any interactions with the author in the course of conducting the reproducibility exercise (help, assistance, clarifications)