NDAs, NDAs, and NDAs: How to describe access to data that cannot be published

I was recently forwarded a question from a data librarian discussion list, and invited to respond to the question of “example language” for a data access description when data were obtained via a non-disclosure agreement (NDA):

[…] the data are from a commercial company under a non-disclosure agreement […] asked to include “data sharing plans (for all data, documentation, and code used in analysis)”. […] Can you think of any example language that the researcher could borrow from to include in the data sharing plan?

(paraphrased)

The use of some sort of data access restriction is quite common in economics. We have a few examples listed here. None of those directly address when an NDA is required, but the generic “confidential data” example is straightforward to adapt.

The data for this project are confidential, but may be obtained with Data Use Agreements with the Massachusetts Department of Elementary and Secondary Education (DESE). Researchers interested in access to the data may contact [NAME] at [EMAIL], also see www.doe.mass.edu/research/contact.html. It can take some months to negotiate data use agreements and gain access to the data. The author will assist with any reasonable replication attempts for two years following publication.

As general guidance, journal (data) editors seek the following information:

• how did the author access the data? (describe how the initial contact was made, what type of agreements, such as NDAs, were signed, and what they restrict and why)
• how can others​ access the data? (“Contact the vice president of academic interaction, and request “data on gooble-dee-gooks” from 2002-2022.”) Importantly, the author should address whether the exact data can be obtained by others, or if this was a pull from a dynamic database, and while generically the same data might be available in a few years’ time, the exact data will never be obtainable. Or that after 10 years, regulatory requirements force the data provider to delete the data. Or whatever is needed to assess the likelihood of being able to access such data. Costs and delays are also important.
• In some cases, authors may already know that the regime within the data provider has changed in such a fashion that the data are no longer available to anybody going forward - that may not be an impediment for the paper being published, but it is important to know (the data are fundamentally unverifiable)

This is summarized in the Data and Code Availability Standard, that many econ journals now adhere to: https://datacodestandard.org/

Examples

Here are some examples (not all exactly corresponding to a single-entity NDA):

Gross & Sampat (2023)

Gross, Daniel P., and Bhaven N. Sampat. 2023. “America, Jump-Started: World War II R&D and the Takeoff of the US Innovation System.” American Economic Review, 113 (12): 3323-56. https://doi.org/10.1257/aer.20221365

• Article

• Replication package

• Some data are included, others are not, because subject to redistribution restrictions, but are obtainable under various agreement, and others are commercially available for payment (thus a variety of DUA types)
• For instance, they state “Derwent Innovation (2018), EPO (2017) PATSTAT, and the Dun &Bradstreet (1980) data were obtained through institutional subscriptions.”
• Excellent data availability statements in the README.

Cullen & Perez-Truglia (2023)

Cullen, Zoë, and Ricardo Perez-Truglia. 2023. “The Old Boys’ Club: Schmoozing and the Gender Gap.” American Economic Review, 113 (7): 1703-40. https://doi.org/10.1257/aer.20210863

• Article

• Replication package

• Data were collected at a company that the first author was an employee of. The company cannot be named (and is pruned from the author’s CV!)
• README states “The data was collected by the authors in collaboration with a large financial institution via surveys. Zoe Cullen was an employee of the institution, which remains unnamed to protect the confidentially of the firm, at the time of data collection.”
• An extreme form of NDA, which happens a few times.

Zussman (2023)

Zussman, Asaf. 2023. “Discrimination in Times of Crises and the Role of the Media.” American Economic Journal: Applied Economics, 15 (4): 422-51. https://doi.org/10.1257/app.20210732

• Article

• Replication package

• The data were scraped from a (named) public website, but contain names of doctors. The data were not made available due to ethical concerns, but the access procedure (repeatable by anybody) is described in detail.
• The author explains that “The project received approval from the Ethics Committee of the Faculty of Social Sciences at the Hebrew University and from the University Ethics Committee (IRB) under the condition that identifying information about the doctors, most importantly doctor license numbers and names, will not become publicly available.”
• Similar: Cage et al (2023) https://doi.org/10.1257/aer.20211509 which contains names of Nazi collaborators.

Kuhn & Shen (2023)

Kuhn, Peter, and Kailing Shen. 2023. “What Happens When Employers Can No Longer Discriminate in Job Ads?” American Economic Review, 113 (4): 1013-48. https://doi.org/10.1257/aer.20211127

• Article

• Replication package

• Internal data from a named Chinese company (XMRC.com) are used, but not published, in “accordance with XMRC’s wishes” (read: NDA/DUA). This is how the authors obtained the data.
• But the limited data used for the analysis are deposited in the Research Data Center of the IZA (a German research center) and can be analyzed on a remote computing platform

Summary

In all of the above, focus was on transparency of the two components of the data availability statement: original access, and subsequent access.

Postscript: Access by the journal is not the same as public access

I will note that in many of these cases, the files were made available to the journal’s Data Editor (=me) under a derivative or implicit NDA, allowing the journal to verify veracity of the analysis, despite the data not being publishable. In cases where the source cannot be named, the Data Editor may enter into an NDA with the author to learn the name of the source, to verify plausibility, without revealing the information publicly. While all of these are second-best access methods, they are far better than obscurity and opacity.

Additional readings

• 2023 post on NDAs
• 2021 post on example of checking confidential data
• AEA policy on third-party replicators
• AEA process for obtaining not-for-distribution data
• Vilhuber, Lars. 2023. “Reproducibility and transparency versus privacy and confidentiality: Reflections from a data editor, Journal of Econometrics, Volume 235, Issue 2, https://doi.org/10.1016/j.jeconom.2023.05.001 (OA version on arXiv)