The following analysis is about all the manuscripts processed since the appointment of Lars Vilhuber as the American Economic Association’s (AEA) inaugural Data Editor in 2018. Processing by the LDI Replication Lab created and run by Lars Vilhuber started slowly in September of 2019, and now handles about 400 articles per year. Any article with a computational component, whether statistical analysis or simulation, is in-scope. Articles published in the Papers and Proceedings receive a cursory review, but are subject to the same policy otherwise.

2389 Manuscripts and 4440 Reports

Overall, the LDI Replication Lab has received evaluation requests for approximately 2389 unique manuscripts. Each round of the manuscript evaluation generates a report. Most papers do no more than 2 rounds. In all, 4440 have been prepared, and resubmitted to the authors, with notes on how to improve the replication packages prior to publication.

Journal Reports
AEA P&P 511
AEJ:Applied Economics 917
AEJ:Economic Policy 600
AEJ:Macro 454
AEJ:Micro 262
AER 943
AER:Insights 244
JEL 130
JEP 212

4378 Authors Reached

Of the approximately 2389 manuscripts, 1480 manuscripts have been published. This excludes articles in the JEP1 and the Papers and Proceedings, which I don’t track for this report.2

For the articles that have been published, I can identify the number of authors associated with these papers, and infer the total number of authors for those articles not tracked or not yet published. Naturally, I cannot assume that every author of the manuscript will have read the Data Editor’s reports, and in that sense, this is an over-estimate of the extent of the impact of the reports. On the other hand, many of the authors have research assistants, tasked with preparing the replication package in the first place, and adjusting it based on the report. In this other sense, the numbers here are an underestimate of both the short-term as well as the long-term impact. Each report is formulated to help authors (and RAs) to improve their ability to publish transparent and credible research.

166 Undergraduates

While most articles published in AEA journals are not published by undergraduates, almost all the reproducibility checks in the LDI Replication Lab are conducted by undergraduates, mostly by students from Cornell, not all of which have a connection to the Economics department. Since the Lab started doing this, every four months, a cohort of undergraduates are recruited and trained. Training is necessary, because many of the conceptual tools (data provenance, objective reporting) and practical tools (running code, using version control) are not in the (explicit) curriculum of almost all these students. Training takes about 20 hours. Students are recruited primarily in their junior year, but I have had several freshmen, and several students have stayed with the Lab until (and sometimes beyond) their graduation date. The skills imparted have, anecdotally, also proven useful in many non-academic (real-world job) environments, such as consulting.

Overall, 166 undergraduate students have been engaged in this academic activity. Their names are listed in the Appendix. In addition, a pilot project in Summer 2024 provided interships to 9 undergraduates from various undergraduate institutions around the U.S.

Graduate Students and Pre-docs

Naturally, this is not a job to manage as a single person. The LDI Replication Lab has had the participation of six graduate students, and three pre-docs. Both prior pre-docs have gone on to do an economics Ph.D. program.

Impact elsewhere

Lars Vilhuber, as the AEA Data Editor, organizes a monthly meeting of social science data editors. These are currently predominantly in economics and management, at journals supported by learned societies (as opposed to journals owned by commercial publishers or run by economics departments). The group includes approximately 10 data editors, who coordinate on processes, and issue joint guidance.

Funding

The AEA Data Editor has a budget to conduct the activities outlined above. All student employment is fully funded as part of that budget, as are the pre-docs. Graduate students’ stipends and health insurance is covered, but tuition is not. However, the budget from the AEA is likely to be cut in the upcoming years, and the current budget cannot be used for more broad-ranging activities, for instance to support similar activities at journals in economics and elsewhere in the empirical social sciences that do not have a similar stature as the AEA. Auxiliary activities, such as the meeting of data editors, are unfunded. The Economics Department and the ILR School make significant in-kind contributions, such as tuition waivers and all the compute time used by the hundreds of reproduction attempts.

Appendix: Undergraduate students

The following students have participated in the LDI Replication Lab, however briefly: Adam J. Faridi, Akshay Yadava, Albert Li, Alice Wei, Alizay Zartash, Amie Li, Ananya Bakshi, Andreas Psahos, Andres Aradillas Fernandez, Andrew Phiri, Andrew Wallace, Angelina Chen, Anjini Khanna, Anthony Peraza, Anurag Tiwari, Arnaav Sareen, Asha Patt, Ashley Cooray, Ashley Yu, Benjamin Zheng, Bianca Jimenez, Botao Yao, Cade Lenczycki, Caitlin Song, Christine Cho, Claire Vaughn, Connor Williams, Craig Schulman, Crystal Lim, Daniella Pena, David Wasser, Dhilan Bansal, Dmitry Shlyapnikov, Edward Vu, Eli Kolodezh, Elian Gomez, Elijah Ruiz, Elliott Serna, Emily Brydges, Emma Sbrollini, Ethan Carlson, Franklin Otieno, Gabriel Bond, Gary Wu, Gavin Bramley, Giles Mitchell, Hana Saleh, Hawi Tolera, Henry Tan, Hongyi Duan, Huey Le, Hyuk Son, Ilanith Nizard, Ilona Khimey, Jack Donnellan, Jacob Brogdon, Jacob Recht, Jade Yang, Jaeyoung Shim, Jai Kishore Kumar Chandnani, Janet Malzahn, Jared Martin, Jason Lan, Jeong Hyun Lee, Jessica Rizzo, Jill Crosby, John Park, Jonah Huang, Jonathan Temkin, Joshua Passell, Joshua Wallace, Julia Zimmerman, Kai Wang, Kareena Stowers, Kate Chanpong, Kate Hofer, Katheryn Ding, Kevin Bao, Kirin Eicher, Kirubeal Wondimu, Kristine Li, Kushal Kumar Reddy, Lauren Stubbs, Leah Kim, Lei Huang, Leslie Geng, Liam Cushen, Lilly Thomalla, LinchenZhang, Lincy Chen, Louis Liu, Luis Lopez Cabrera, Luke Trautwein, Lydia Reiner, Manas Gogula, Manvir Chahal, Marina Starikovsky, Mary-Jo Ajiduah, Mateo Pesa, Matt Wang, Matthew LaFontaine, Maximilian Yap, Melanie Brown, Melanie Chen, Meredith Welch, Micere Mugweru, Milena Zhu, Miranda Zhou, Mona Wu, Nahid Hassan, Naomi Li, Nathan Maidi, Nehedin Juarez, Nguyen Vo, Nicholas Esarte, Nicholas Swan, Nishat Tasneem Peuly, Olivia Liu, Ololade Omotoba, Peter Sanchez, Phalguni Miraj, Qianyi Liu, Raymond Wang, Rubal Mistry, Ryan Ali, Sam Evans, Satya Datla, Sean Miranda, Seong Hwan Kim, Sharan Banerjee, Sharon Liu, Sherry Li, Siddhi Malvankar, Siyang Elaine Yu, Sohit Gurung, Sourabh Velaga, Steve Yeh, Surita Basu, Suvd Khaliun, Sylverie Herbert, Syon Verma, Talia Boehm, Tarangana Thapa, Taren Daniels, Tommy Wang, Tony Ford Jr, Tyler Irving, Valerie Setiawan, Vansh Gupta, Vedha Mahesh, Victoria Liu, Vidya Balaji, Weilun Shi, Weiting Shen, Xiangru Li, Ximei Shen, Yanyun Chen, Yicheng Yang, Yuchang Tian, Zebang Xu, Zechariah Karsana, Kayla Yang, Jiayin Song, Nick Cichoski, Sharon Lin, Zhaojiahong Zhu.

  1. I cannot track papers in the Journal of Economic Perspective, since our internal database only tracks manuscript numbers, which are cannot be simply mapped into DOIs. The LDI Replication Lab’s database does not track the actual DOI of the published manuscript, as that is handled by the AEA’s editorial office.↩︎

  2. Papers and Proceedings receive a much more cursory review.↩︎