The following analysis is about all the manuscripts processed since the appointment of Lars Vilhuber as the American Economic Association’s (AEA) inaugural Data Editor in 2018. Processing by the LDI Replication Lab created and run by Lars Vilhuber started slowly in September of 2019, and now handles about 400 articles per year. Any article with a computational component, whether statistical analysis or simulation, is in-scope. Articles published in the Papers and Proceedings receive a cursory review, but are subject to the same policy otherwise.

3244 Manuscripts and 5064 Reports

Overall, the LDI Replication Lab has received evaluation requests for approximately 3244 unique manuscripts. Each round of the manuscript evaluation generates a report. Most papers do no more than 2 rounds. In all, 5064 have been prepared, and resubmitted to the authors, with notes on how to improve the replication packages prior to publication.

Journal Reports
AEA P&P 696
AEJ:Applied Economics 1070
AEJ:Economic Policy 684
AEJ:Macro 546
AEJ:Micro 284
AER 1104
AER:Insights 285
JEL 133
JEP 199
Other journals 8

5605 Authors Reached

Of the approximately 3244 manuscripts, 670 are articles in the Papers and Proceedings, which receive a different report.1 Of the remaining manuscripts, 2066 manuscripts have been published. This count does not include the 150 articles in the JEP, which I cannot track.2

For the articles that have been published, I can identify the number of authors associated with these papers, and infer the total number of authors for those articles not tracked (JEP) or not yet published, using the average number of authors per published article. Naturally, I cannot assume that every author of the manuscript will have read the Data Editor’s reports, and in that sense, this is an over-estimate of the extent of the impact of the reports. On the other hand, many of the authors have research assistants, tasked with preparing the replication package in the first place, and adjusting it based on the report. In this other sense, the numbers here are an underestimate of both the short-term as well as the long-term impact. Each report is formulated to help authors (and RAs) to improve their ability to publish transparent and credible research.

239 Undergraduates

While most articles published in AEA journals are not published by undergraduates, almost all the reproducibility checks in the LDI Replication Lab are conducted by undergraduates, mostly by students from Cornell, not all of which have a connection to the Economics department. Since the Lab started doing this, every four months, a cohort of undergraduates are recruited and trained. Training is necessary, because many of the conceptual tools (data provenance, objective reporting) and practical tools (running code, using version control) are not in the (explicit) curriculum of almost all these students. Training takes about 20 hours. Students are recruited primarily in their junior year, but I have had several freshmen, and several students have stayed with the Lab until (and sometimes beyond) their graduation date. The skills imparted have, anecdotally, also proven useful in many non-academic (real-world job) environments, such as consulting.

Overall, 239 undergraduate students have been engaged in this academic activity. Their names are listed in the Appendix. In addition, since the Summer of 2024, I have provided interships to 9 undergraduates each year from various undergraduate institutions around the U.S.

Graduate Students and Pre-docs

Naturally, this is not a job to manage as a single person. The LDI Replication Lab has had the participation of 10 graduate students, and 3 pre-docs. Both prior pre-docs have gone on to do an economics Ph.D. program.

Impact elsewhere

Lars Vilhuber, as the AEA Data Editor, organizes a monthly meeting of social science data editors. These are currently predominantly in economics and management, at journals supported by learned societies (as opposed to journals owned by commercial publishers or run by economics departments). The group includes approximately 10 data editors, who coordinate on processes, and issue joint guidance.

Funding

The AEA Data Editor has a budget to conduct the activities outlined above. All student employment is fully funded as part of that budget, as are the pre-docs. Graduate students’ stipends and health insurance is covered, but tuition is not. However, the budget from the AEA is likely to be cut in the upcoming years, and the current budget cannot be used for more broad-ranging activities, for instance to support similar activities at journals in economics and elsewhere in the empirical social sciences that do not have a similar stature as the AEA. Auxiliary activities, such as the meeting of data editors, are unfunded. The Economics Department and the ILR School make significant in-kind contributions, such as tuition waivers and all the compute time used by the hundreds of reproduction attempts.

Appendix: Undergraduate students

The following students have participated in the LDI Replication Lab, however briefly: Adam J. Faridi, Adeline Sutton, Adis Bajraktarevic, Adrik Aburto, Aishah Abdul-Hakeem, Akshay Yadava, Albert Li, Alice Wei, Alizay Zartash, Amber Wang, Amie Li, Amoghadithya Divya Swaroop, Ananya Bakshi, Andreas Psahos, Andres Aradillas Fernandez, Andrew Phiri, Andrew Wallace, Angelina Chen, Anita Putri Wulandari, Anjini Khanna, Ankriti Singh, Anna Sundheim, Anthony Peraza, Anurag Tiwari, Araf Malik, Arnaav Sareen, Aryan Shah, Asha Patt, Ashley Cooray, Ashley Yu, Austin Grattan, Benjamin Zheng, Bethelhem Salehu, Bianca Jimenez, Botao Yao, Cade Lenczycki, Caitlin Song, Cameron Kang, Carissa Zheng, Caroline Ciaschini, Christina Chen, Christine Cho, Claire Vaughn, Claron Wang, Connor Williams, Craig Schulman, Crystal Lim, Dane Clunk, Daniel Hurtado, Daniel McGuire, Daniella Pena, David Graham, Dengqi Chen, Dhilan Bansal, Dmitry Shlyapnikov, Edward Vu, Eirdeena Zakirah Mohd Nor Al-Azzad, Eli Kolodezh, Elian Gomez, Elijah Ruiz, Elizabeth Chen, Elliott Serna, Emily Brydges, Emily Chan, Emma Sbrollini, Esther Park, Ethan Carlson, Farhan Mashrur, Franklin Otieno, Gabriel Bond, Gary Wu, Gavin Bramley, Gemie Sonamai, Giles Mitchell, Grace Fang, Grace Yichi Zhang, Hana Saleh, Haomin Yu, Hawi Tolera, Henry Tan, Hongyi Duan, Huey Le, Ian Joyce, Ilona Khimey, Ivor Mills, Jack Donnellan, Jacob Brogdon, Jacob Recht, Jade Yang, Jaeyoung Shim, Jai Kishore Kumar Chandnani, Janet Malzahn, Jared Martin, Jason Lan, Jeong Hyun Lee, Jessica Rizzo, Jesus Robles, Jiatian Yin, Jill Crosby, John Park, Jonah Huang, Jonathan Temkin, Joshua Passell, Joshua Wallace, Joyce Han, Julia Zimmerman, Kai Wang, Kareena Stowers, Kate Chanpong, Kate Hofer, Katheryn Ding, Katie Chen, Kevin Bao, Kirin Eicher, Kirubeal Wondimu, Kiyam Merali, Kristine Li, Kushal Kumar Reddy, Laurel Krovetz, Lauren Stubbs, Leah Kim, Lei Huang, Leslie Geng, Liam Cushen, Lilly Thomalla, LinchenZhang, Lincy Chen, Linh Nguyen, Lorraine Yang, Louis Liu, Luis Lopez Cabrera, Luke Trautwein, Luyanda Chibwe, Lydia Reiner, Manas Gogula, Manvir Chahal, Marina Starikovsky, Mary-Jo Ajiduah, Mateo Pesa, Matt Wang, Matthew LaFontaine, Max Walder, Maximilian Yap, Maya Schmaling, Melanie Brown, Melanie Chen, Micere Mugweru, Milena Zhu, Miranda Zhou, Mohammad Alrahmah, Molly Rabb-Jaros, Mona Wu, Nahid Hassan, Nancy Qiao, Naomi Li, Natalie Miller, Nathan Kwon, Nathan Maidi, Nehedin Juarez, Nguyen Vo, Nicholas Esarte, Nicholas Silvers, Nicholas Swan, Nikhil Vudattu, Nimba Mahlati, Nishat Tasneem Peuly, Olivia Kim, Olivia Liu, Ololade Omotoba, Owen Kinney, Owen Ryan, Peter Sanchez, Phalguni Miraj, Pranav Jaganathan, Qianyi Liu, Ratchaphon Lertdamrongwong, Raymond Wang, Rishik Gowrishetti, Rishika Cherivirala, River Strumwasser, Roman Shelley, Rubal Mistry, Ruhi Khanna, Ryan Ali, Sabrina Gerson, Sam Evans, Sana Khan, Sanjana Bajaj, Sanskriti Giri, Satya Datla, Sean Miranda, Seong Hwan Kim, Sharan Banerjee, Sharon Liu, Sherry Li, Shubham Singh, Siddhi Malvankar, Sidera Hilt, Siyang Elaine Yu, Sohit Gurung, Sonia Zhang, Sourabh Velaga, Steve Yeh, Surita Basu, Suvd Khaliun, Syon Verma, Talia Boehm, Talia Richmond, Tarangana Thapa, Taren Daniels, Tommy Wang, Tony Ford Jr, Tyler Irving, Valerie Setiawan, Vansh Gupta, Vedha Mahesh, Venna Huang, Victoria Liu, Vidya Balaji, Vivian Ye, Weilun Shi, Weiting Shen, William Howard, Wyatt Lewis, Xiangru Li, Ximei Shen, Yanyun Chen, Yicheng Yang, Yuchang Tian, Zachary Mangoba, Zebang Xu, Zechariah Karsana, Zoey Yu, Kayla Yang, David Fenner, Jiayin Song, Nick Cichoski, Sharon Lin, Zhaojiahong Zhu.

  1. Papers and Proceedings receive a much more cursory review, and their authors are counted separately below.↩︎

  2. I cannot track papers in the Journal of Economic Perspectives, since our internal database only tracks manuscript numbers, which cannot be simply mapped into DOIs. The LDI Replication Lab’s database does not track the actual DOI of the published manuscript, as that is handled by the AEA’s editorial office.↩︎