The code and data that have been archived should be reproducible and replicable. How do we test that?
Generic guidance is provided at the Social Science Data Editors’ Guidance website.
At the AEA
We assess - software availability - data availability - code availability and clarity - needed computational resources - time needed to acquire or use all of the above, and conduct the reproducibility check
What if we don’t have the resources
When some of the conditions are not met with our own resources, we may ask others to conduct a reproducibility exercise for us. - Our PROTOCOL describes how we might request reports. - Sample report 1 - We may ask others to do so because - They are experts - They have access to the software - They have access to the data - They have access to computational resources needed
- No reproducibility check is discarded out of hand