The following analysis is about all the manuscripts processed since the appointment of Lars Vilhuber as the American Economic Association’s (AEA) inaugural Data Editor in 2018. Processing by the LDI Replication Lab created and run by Lars Vilhuber started slowly in September of 2019, and now handles about 400 articles per year. Any article with a computational component, whether statistical analysis or simulation, is in-scope. Articles published in the Papers and Proceedings receive a cursory review, but are subject to the same policy otherwise.

2791 Manuscripts and 4470 Reports

Overall, the LDI Replication Lab has received evaluation requests for approximately 2791 unique manuscripts. Each round of the manuscript evaluation generates a report. Most papers do no more than 2 rounds. In all, 4470 have been prepared, and resubmitted to the authors, with notes on how to improve the replication packages prior to publication.

Journal Reports
AEA P&P 573
AEJ:Applied Economics 974
AEJ:Economic Policy 609
AEJ:Macro 485
AEJ:Micro 258
AER 967
AER:Insights 260
JEL 117
JEP 182

4973 Authors Reached

Of the approximately 2791 manuscripts, 1724 manuscripts have been published. This excludes articles in the JEP1 and the Papers and Proceedings, which I don’t track for this report.2

For the articles that have been published, I can identify the number of authors associated with these papers, and infer the total number of authors for those articles not tracked or not yet published. Naturally, I cannot assume that every author of the manuscript will have read the Data Editor’s reports, and in that sense, this is an over-estimate of the extent of the impact of the reports. On the other hand, many of the authors have research assistants, tasked with preparing the replication package in the first place, and adjusting it based on the report. In this other sense, the numbers here are an underestimate of both the short-term as well as the long-term impact. Each report is formulated to help authors (and RAs) to improve their ability to publish transparent and credible research.

201 Undergraduates

While most articles published in AEA journals are not published by undergraduates, almost all the reproducibility checks in the LDI Replication Lab are conducted by undergraduates, mostly by students from Cornell, not all of which have a connection to the Economics department. Since the Lab started doing this, every four months, a cohort of undergraduates are recruited and trained. Training is necessary, because many of the conceptual tools (data provenance, objective reporting) and practical tools (running code, using version control) are not in the (explicit) curriculum of almost all these students. Training takes about 20 hours. Students are recruited primarily in their junior year, but I have had several freshmen, and several students have stayed with the Lab until (and sometimes beyond) their graduation date. The skills imparted have, anecdotally, also proven useful in many non-academic (real-world job) environments, such as consulting.

Overall, 201 undergraduate students have been engaged in this academic activity. Their names are listed in the Appendix. In addition, since the Summer of 2024, I have provided interships to 9 undergraduates each year from various undergraduate institutions around the U.S.

Graduate Students and Pre-docs

Naturally, this is not a job to manage as a single person. The LDI Replication Lab has had the participation of seven graduate students, and three pre-docs. Both prior pre-docs have gone on to do an economics Ph.D. program.

Impact elsewhere

Lars Vilhuber, as the AEA Data Editor, organizes a monthly meeting of social science data editors. These are currently predominantly in economics and management, at journals supported by learned societies (as opposed to journals owned by commercial publishers or run by economics departments). The group includes approximately 10 data editors, who coordinate on processes, and issue joint guidance.

Funding

The AEA Data Editor has a budget to conduct the activities outlined above. All student employment is fully funded as part of that budget, as are the pre-docs. Graduate students’ stipends and health insurance is covered, but tuition is not. However, the budget from the AEA is likely to be cut in the upcoming years, and the current budget cannot be used for more broad-ranging activities, for instance to support similar activities at journals in economics and elsewhere in the empirical social sciences that do not have a similar stature as the AEA. Auxiliary activities, such as the meeting of data editors, are unfunded. The Economics Department and the ILR School make significant in-kind contributions, such as tuition waivers and all the compute time used by the hundreds of reproduction attempts.

Appendix: Undergraduate students

The following students have participated in the LDI Replication Lab, however briefly: Adam J. Faridi, Adis Bajraktarevic, Aishah Abdul-Hakeem, Akshay Yadava, Albert Li, Alice Wei, Alizay Zartash, Amie Li, Ananya Bakshi, Andreas Psahos, Andres Aradillas Fernandez, Andrew Phiri, Andrew Wallace, Angelina Chen, Anjini Khanna, Anthony Peraza, Anurag Tiwari, Arnaav Sareen, Aryan Shah, Asha Patt, Ashley Cooray, Ashley Yu, Austin Grattan, Benjamin Zheng, Bianca Jimenez, Botao Yao, Cade Lenczycki, Caitlin Song, Cameron Kang, Carissa Zheng, Christine Cho, Claire Vaughn, Connor Williams, Craig Schulman, Crystal Lim, Dane Clunk, Daniella Pena, David Graham, David Wasser, Dhilan Bansal, Dmitry Shlyapnikov, Edward Vu, Eirdeena Zakirah Mohd Nor Al-Azzad, Eli Kolodezh, Elian Gomez, Elijah Ruiz, Elizabeth Chen, Elliott Serna, Emily Brydges, Emma Sbrollini, Esther Park, Ethan Carlson, Farhan Mashrur, Franklin Otieno, Gabriel Bond, Gary Wu, Gavin Bramley, Gemie Sonamai, Giles Mitchell, Grace Fang, Grace Yichi Zhang, Hana Saleh, Hawi Tolera, Henry Tan, Hongyi Duan, Huey Le, Hyuk Son, Ilona Khimey, Jack Donnellan, Jacob Brogdon, Jacob Recht, Jade Yang, Jaeyoung Shim, Jai Kishore Kumar Chandnani, Janet Malzahn, Jared Martin, Jason Lan, Jeong Hyun Lee, Jessica Rizzo, Jesus Robles, Jiatian Yin, Jill Crosby, John Park, Jonah Huang, Jonathan Temkin, Joshua Passell, Joshua Wallace, Julia Zimmerman, Kai Wang, Kareena Stowers, Kate Chanpong, Kate Hofer, Katheryn Ding, Kevin Bao, Kirin Eicher, Kirubeal Wondimu, Kristine Li, Kushal Kumar Reddy, Lauren Stubbs, Leah Kim, Lei Huang, Leslie Geng, Liam Cushen, Lilly Thomalla, LinchenZhang, Lincy Chen, Lorraine Yang, Louis Liu, Luis Lopez Cabrera, Luke Trautwein, Lydia Reiner, Manas Gogula, Manvir Chahal, Marina Starikovsky, Mary-Jo Ajiduah, Mateo Pesa, Matt Wang, Matthew LaFontaine, Max Walder, Maximilian Yap, Melanie Brown, Melanie Chen, Meredith Welch, Micere Mugweru, Milena Zhu, Miranda Zhou, Mona Wu, Nahid Hassan, Naomi Li, Natalie Miller, Nathan Kwon, Nathan Maidi, Nehedin Juarez, Nguyen Vo, Nicholas Esarte, Nicholas Swan, Nishat Tasneem Peuly, Olivia Kim, Olivia Liu, Ololade Omotoba, Owen Kinney, Owen Ryan, Peter Sanchez, Phalguni Miraj, Pranav Jaganathan, Qianyi Liu, Ratchaphon Lertdamrongwong, Raymond Wang, Rishika Cherivirala, River Strumwasser, Rubal Mistry, Ryan Ali, Sam Evans, Sana Khan, Sanjana Bajaj, Satya Datla, Sean Miranda, Seong Hwan Kim, Sharan Banerjee, Sharon Liu, Sherry Li, Siddhi Malvankar, Siyang Elaine Yu, Sohit Gurung, Sonia Zhang, Sourabh Velaga, Steve Yeh, Surita Basu, Suvd Khaliun, Sylverie Herbert, Syon Verma, Talia Boehm, Talia Richmond, Tarangana Thapa, Taren Daniels, Tommy Wang, Tony Ford Jr, Tyler Irving, Valerie Setiawan, Vansh Gupta, Vedha Mahesh, Venna Huang, Victoria Liu, Vidya Balaji, Vivian Ye, Weilun Shi, Weiting Shen, William Howard, Wyatt Lewis, Xiangru Li, Ximei Shen, Yanyun Chen, Yicheng Yang, Yuchang Tian, Zebang Xu, Zechariah Karsana, Kayla Yang, Jiayin Song, Nick Cichoski, Sharon Lin, Zhaojiahong Zhu.

  1. I cannot track papers in the Journal of Economic Perspective, since our internal database only tracks manuscript numbers, which are cannot be simply mapped into DOIs. The LDI Replication Lab’s database does not track the actual DOI of the published manuscript, as that is handled by the AEA’s editorial office.↩︎

  2. Papers and Proceedings receive a much more cursory review.↩︎