Guidance on testing replicability of code


The code and data that have been archived should be reproducible and replicable. How do we test that?

Generic guidance

Generic guidance is provided at the Social Science Data Editors’ Guidance website.

At the AEA

We use this template to guide our replicators. - Example 1

We assess - software availability - data availability - code availability and clarity - needed computational resources - time needed to acquire or use all of the above, and conduct the reproducibility check

What if we don’t have the resources

When some of the conditions are not met with our own resources, we may ask others to conduct a reproducibility exercise for us. - Our PROTOCOL describes how we might request reports. - Sample report 1 - We may ask others to do so because - They are experts - They have access to the software - They have access to the data - They have access to computational resources needed

  • No reproducibility check is discarded out of hand