From a recent (draft) replication package:

Data on Kelly et al. (2021)’s patent indicator are downloaded from the authors’ GitHub repository. A copy of the data is provided as part of this archive. The data are in the public domain. To download, visit at https://github.com/KPSS2017, click the pinned repository named “Measuring-Technological-Innovation-Over-the-Long-Run-Extended-Data,” and download “PatentSimilarityImportanceBreakthrough_forPost2022.csv.zip”.

A few things, first for those re-using the data, and then for those providing the data.

For re-users

Citing

If using the Github data, and not the replication package to the original article, the Github data must be cited. Here, that would be

Kelly, B., Papanikolaou, D., Seru, A. and Taddy, M. 2023. Updates and Extension of Measuring Technological Innovation Over the Long Run. https://github.com/KPSS2017/Measuring-Technological-Innovation-Over-the-Long-Run-Extended-Data, version from November 2023.

On the Github README, the authors actually say

The version released on Sept 29, 2023 is the latest.

but the commits on the Github actually tell a different story: there are September and November updates to the deposit. Whether those are materially different is not clear (these are ZIP files), but as a re-user, you should identify the version you ACTUALLY used.

Note that the authors ask that their published article be cited as a source. That is fine, as long as the re-user ALSO cites the actual source of the data, which is Github.

Downloading

The very verbose description is actually not the best way. First, the actual Github repository has a direct link. The pinned repositories can change over time, without any reason. Re-users should not rely on that. In fact, as I mentioned above, there are multiple commits. Unless the Github owners manipulate those commits (possible), the specific commit is what should be identified, possibly in the citation, but definitely in the download instructions. Here is the same verbose instructions expressed as a single URL:

Download https://github.com/KPSS2017/Measuring-Technological-Innovation-Over-the-Long-Run-Extended-Data/blob/42f283517d72d90272f151867bb7b22744e23b2f/PatentSimilarityImportanceBreakthrough_forPost2022.csv.zip

which preserves the metadata (it doesn’t directly download the data, but rather, allows you to see the context in which you are downloading the data). Much simpler.

Distributing

The quote I provided mentions “data are in the public domain.” That is incorrect. The data are downloadable by anybody without registration. However, “public domain” is a legal term that means that nobody has the copyright on the data. Possibly because the entity producing the data is legally excluded from claiming copyright (oversimplifying somewhat, in the US, the federal government), or because the person producing the data has explicitly relinquished this (often under a “CC0” license).

But in the absence of true “public domain”, the data are copyrighted by the original authors. Fair-use is fine (so using the data is likely the intent of posting it on the internet here), but re-purposing the data may not be. Copyright = all rights reserved does not need an explicit mention! So strictly speaking, you need the authors’ permission to redistribute the data, or you might have to claim “fair use” explicitly. We usually send the re-users to ask permission, which is cleaner.

For authors

All of the above can be greatly simplified if authors (those posting on Github) followed some simple best practices:

Define a license

Authors should clearly specify the license under which the data are released. Common licenses include Creative Commons (CC) licenses, which allow for various levels of reuse of data, or open-source software licenses like MIT or GPL, which might be more relevant for code. A dual-license setup can work. See guidance on my website. The license should be expressed as a LICENSE.txt or LICENSE.md file in the root of the repository, it will be picked up and displayed in the sidebar by Github.

If authors do NOT want to provide a general license, that is also fine, but they should likely explicitly say it, to be clear, although that may not be legally required: “Copyright John Doe and Alice Smith, all rights reserved. Permission to redistribute or reuse this data must be obtained from the authors.”

Versioning

Rather than having re-users guess at versions, and have long commit-based URLs, create tags or releases. Very easy to do in both git and on Github. That would allow for much simpler URLs:

Download https://github.com/KPSS2017/Measuring-Technological-Innovation-Over-the-Long-Run-Extended-Data/blob/ v1.1 /PatentSimilarityImportanceBreakthrough_forPost2022.csv.zip

(OK, that’s not much shorter in this case, because of the long repository and file names, but it is easier to read).

Citation

Authors in general prefer that you cite the article, because under current academic norms (at least in economics), is the only thing that counts. Nevertheless, if the article is published in 2021, and the data in 2023, re-users are misleading their readers because they materially reference data that does not exist at the cited location (the article, or its replication package), and cannot be easily linked. The solution is to cite both.

And to make that easy, use the CITATION.cff file. It, too, is highlighted by Github in the right panel.

Preserving

And finally, if you have license and citation all tidied up, why not preserve it, and get a DOI to boot? This does away with the fragility of deposits and commits (anybody can delete their Github repo forever, in 30 seconds flat), and simplifies citation as well. You can do so nearly automatically (if creating releases) via Zenodo, or to Dataverse.

Re-users can then simply use the Zenodo version, which has a DOI, and is guaranteed to be preserved.

First: do not hide the output!

The immediate problem was not the regression itself. It was that we could not see the regression output, and thus the error.

The offending line was, in essence:

quietly do regression_55.do

quietly suppressed exactly the output that would have explained the failure. The code simply stopped completing, with nothing on screen to say why. Diagnosing it meant unwinding the suppression before we could even start on the econometrics.

It’s not exactly the code

After some back and forth, the authors provided (credible) evidence that the code did run on their system. So we reduced their several hundred lines to a minimal working example of about two dozen, and sent it to Stata support, who responded promptly!

It is not the operating system

Our first hypothesis was the obvious one. We had run on Windows Server (StataNow 19) and Linux (StataNow 19); the authors had run on Windows Enterprise (Stata 19). Operating system, then. Or version.

That was not enough. Stata’s technical staff ran the example across their internal collection of hardware and software, and reported:

convergence on Windows box running Stata 18 MP32 and Stata 19 MP8, but set processors to any number less than 8 cause not converging….Mac mini Apple M2 pro/ S19MP16 NOT…Linux/S19MP16, S19SE, S18MP2, S17MP2, S16MP2, NOT converged for any number of processors

One should take particular note of this fact: On the same Windows machine, with the same Stata, the model converges or does not converge depending on how many processors you let it use. The authors, it turned out, had run Stata 19 MP10 — one of the few combinations in which the model converged at all.

The determining variable was not the operating system, and not the Stata version. It was the number of processors.

What was actually wrong

Stata’s statisticians examined the code and the data, and the answer is that it’s not “a bug.” Mostly. Quoting from their reply:

Stata, like any other software package, has fixed criteria to convergence declaration. It’s not foolproof. There will be always cases where convergence will be “declared” and it shouldn’t.

This is why we insist on the importance of examining the output and make sure that it makes sense. I looked at the output it shows several problems.

Three symptoms, all visible in the output that quietly had been hiding:

1) there are missing standard errors. … an indication that there is a problem with the Hessian (either it has missing values, or for some reason it can’t be inverted), so you can’t trust your results.

2) There are huge confidence intervals,… coefficients are poorly identified.” 3) The problem is the matrix of second derivatives, that is non invertible. In the cases where we get ‘convergence’, we get huge or missing standard errors, meaning that it can’t really be inverted.”

The mechanism is worth spelling out, because it explains the platform sensitivity. Stata declares convergence when the scaled gradient satisfies g*inv(H)*g' < nrtolerance(), with a default of 1e-5. That criterion requires inverting the Hessian. When the Hessian is numerically singular, the criterion cannot be evaluated in a meaningful way — and whether it happens to be evaluated as satisfied comes down to floating-point details that differ with the number of processors, the BLAS in use, and the order in which sums are accumulated (my words, not theirs).

They continued.

On Mac, I don’t get convergence by default, but I get it if I disable the nrtolerance, meaning that this is the tolerance that the model fails to meet.

If I write shownrtol option instead, I will see that (the “minimum” of all tolerances) is 0, but exact zero means that something can’t be computed, in this case the nrtol (which requires inverting the Hessian).

So the model was not identified on any platform. On most configurations Stata correctly refused to declare victory. On a few — including the authors’ — it declared convergence it should not have.

Why this matters beyond one paper

Two observations:

This was one regression out of many dozens in the paper. The rest converged. How well they converged we did not investigate — that remains for the authors, and for future replicators.

More importantly, consider what would have happened had our replication machines been configured slightly differently. We only detect some of these issues. If we happen to land on the same sweet-spot hardware and software configuration as the authors, we never see a discrepancy, and the paper passes. The non-identification is still there; we just never trip over it. And it is genuinely hard for authors to diagnose on their own, unless they happen to maintain the same museum of hardware and software versions that Stata does.

Lessons

I will not pronounce on the econometrics — that is the authors’ problem, and for future replicators (who have a reproducible package to work with!) But two things are general:

Know your regressions. Missing standard errors and implausibly wide confidence intervals are not cosmetic blemishes to be tidied up in the table. They are the software telling you the model is not identified. Look at the output.

Do not suppress your output. In Stata, that means thinking twice about quietly. The equivalents elsewhere are just as effective at hiding a problem: source() without echo=TRUE in R, or a stray ; in MATLAB. Silence during a long run feels tidy. It is not tidy — it is a diagnostic you have thrown away, and you will want it back at precisely the moment it is gone.

Thanks to @stata.bsky.social, whose helpdesk responded quickly and then very thoroughly, and did a fantastic job diagnosing this.

Good night, and good computing!

This post is adapted from a thread originally posted on Bluesky.

Of course, it is the first claim a fraudster would make, too. So how can we help legitimate authors separate themselves from the fraudsters, and still publish interesting and real research in journals?¹

The AEA’s policy on private data

The AEA Data and Code Availability Policy asks that authors provide access to all data used in a published paper. But it also acknowledges that there may be cases where data cannot be shared.

In rare cases where any of requirements (i)-(iv) above cannot be met due to valid constraints beyond the authors’ control, the AEA data editor and the journal editor may choose to modify the requirement(s), and such modifications will be noted in the article acknowledgements. However, in these cases the source of the data must still be disclosed to the AEA data editor prior to final acceptance of the paper.

How often does this happen?

A legitimate question is: does this happen often? The answer is no. Upon ingest, data access is classified into five categories: included (fully open data), and four categories of restricted access, going from “very easy to access” to “very difficult”.²

The “anonymous data” obviously falls into the last category, and in the last twelve months, we’ve had 5 such cases.

What do we then do?

Data provision

In a nutshell, our first preference is that authors are allowed to share the data with us, but not-for-publication. This means that we will have access to the data, and can run code against the data, but won’t publish the data.³

We provide a secure upload link, different from the draft deposit. Readers can view our template email with which we reach out to authors at here. Of course, authors should check with their data provider if provision of data is OK. We can also sign a non-disclosure agreement (NDA) if necessary (referenced in the email), but we stand by our word, and most authors take us up on that.

Name of the provider

For the name of the company, we may sign a separate NDA, to protect both us and the authors. Again, we have an SOP (Standard Operating Procedure) for that; here’s the (separate) template email for that. Important: the data provider should be aware of us signing the NDA, as we will publish the fact that the authors revealed the name to me.

Verification

Usually, I then contact the data provider, or a representative of the provider, directly. I keep no records of identity or other means of contact, and usually make video or phone calls, possibly using WhatsApp or Signal. I validate the representative’s identity through independent means (are they really who they claim to be). I ask questions regarding the authors’ access to the data: Did they actually get access? How? Can others get access? Why is the provider’s identity meant to be secret?

Once I have satisfied my professional curiosity, I report back to the editor in charge of the paper, and convey my opinion regarding the legitimacy of the data provider and of the data. The editor then makes a decision on whether to proceed with publication. Once we have obtained the data, we proceed to verify computational reproducibility. We will also work with the authors to make the restrictions and possibilities clear in the README.

Publication

Once all the other pieces fall into place, and the article is published, readers can find this information (for now) in the title page footnote:

“As part of the reproducibility checks conducted by the AEA, the identity of the anonymous data provider was revealed to the Data Editor, who verified legitimacy of the data use agreement and of access to the data.”

An example (albeit membership-gated) is

Haegele, Ingrid. 2026. “Talent Hoarding in Organizations.” American Economic Review 116 (8): 3110–51. https://doi.org/10.1257/aer.20220264

Closing remarks

Our process is not perfect, and probably there are ways to game it. We don’t reveal all the checks that we do, which may also include checking the data content, and for consistency with public data. But we hope that this allows the readers to have some confidence in the research being presented. The authors, of course, should also be sure to write their publication in such a way that it can be replicated in other contexts. After all, their access to a single anonymous firm, city, municipality, etc. remains a “N of 1” case, very interesting, but in need of generalization. The paper and the replication package hopefully help with that.

The AEA Data and Code Availability Policy asks that authors provide access to all data used in a published paper. But it also acknowledges that there are cases where data cannot be shared. This may be due to privacy concerns, proprietary rights to the data that are not the author’s, or terms of use that prohibit the sharing, redistribution, or publication of data.

In principle, Personally Identifiable Information (PII) should never be included in replication packages. There are multiple checks for this. Since the start of creating reports, we would (manually, or semi-automatically) run PII checks (based on J-PAL’s PII_stata_scan.do, and the replicator would add notes on that. More recent reports (since 2024) persistently include text alerting authors to check their data, regardless of the outcome of the check. When depositing data, authors are required to assert that individuals cannot be identified.

But… it happens. Authors believe that their team has removed all PII, and yet some leaks in. Sometimes, it’s because they forgot to remove the verbatim answer to “who takes care of your child when you are at work”. Sometimes, it’s the IP address of the respondent.

So what happens when an author inadvertently does share such data, and now needs to remove it? As a reminder, the author is committed (by signing an agreement) to not withdrawing the replication package. How to proceed? Read on!

In a nutshell

The gist of the solution is the following, for AEA publications.

• The author requests that the deposit (in the AEA case, ICPSR) unpublish the original replication package (call it V1) (also called “de-accessioning”), i.e., remove the ability to download the package. The journal (data editor, me) should be notified at the same time.
• Remove the infringing data from V1 of the replication package.
• [Ideally] Create a new, separate deposit that is compliant with IRB restrictions on PII. I expand on this below.
• Document the new location of the now absent data in the README of the replication package, and how somebody can access it.
• Revise your V1 replication package to take into account changed location, or coding of variables, as per our Policy on Revisions of Data and Code Deposits in the AEA Data and Code Repository.
• Submit the revised replication package to the journal.
• The journal publishes V2 of the replication package, which is now compliant.

Important notes:

• At no time should any code be removed from the package. If at all, minimal changes should be made.
• It is never acceptable to simply remove the old package, without handling replacement.

Preliminary steps

The first steps involve getting access to your own replication package. This may differ by when the replication package was published.¹

• For authors with papers published before 2019 (approximately)
  • All authors will need to get an openICPSR login, which will serve them both for the changes to the AEA and for creating the new PSID deposit.
  • They will need to request access to the deposit. The openICPSR helpdesk may be able to help with that. The authors should mention the URL or DOI, and the openICPSR login they want the deposit associated with.

• For authors with more recent papers:

  • They will need to re-create an ICPSR account with the same email as previously used since ICPSR changed their login system in Feb 2024. The openICPSR helpdesk may help with that.
  • They need to then create a new version.

]

Rules that apply to revisions of replication packages

Here are a few simple rules:

• The modified package should differ only as necessary from the original package. Less work for the author, greater clarity for replicators, everybody wins.
• Journal-specific policies may apply. For instance, the AEA has a Policy on Revisions of Data and Code Deposits in the AEA Data and Code Repository. Among other things, it requires that all changes be clearly identified.

Fixing the problem: where to put the data?

Once the preliminaries are done, here’s what authors need to do to remain compliant with AEA policy (similar rules may apply at other journals):

• Consider how to preserve the PII-laden data.

Ideally, you already have a copy of the data, to be compliant with AEA policy (5 years post-publication). There are options for preserving the data at your institution, or archives like ICPSR, but they vary over time. Talk to your data librarian, or to data custodians at ICPSR. Talk to the Data Editor.

• The AEA prefers strongly that such deposits NOT require the authors’ consent to access.

Rather, access should be granted based on clear terms of use (“provide IRB approval”), vetted by individuals not connected to the author, and likely to be present long after the author has retired. Institutions, like your own university or ICPSR, are ideal.

Fixing the problem: what data to include?

If the variables that contain PII are not essential to the replication of the paper, then they can simply be removed from the replication package. This must still be documented!

If however, they are important to the analysis (left-hand side or right-hand side), you will need to be more creative. You may document that certain analyses are no longer feasible without the PII-laden, but should ensure that the code continues to run for all other analyses (for instance, because the PII-laden variables only affect one out of a gazillion robustness checks).

You may also be able to recode the PII, replacing them with innocuous but still unique identifiers. Finally, you may need to replace variables with fake or synthetic data, which will ensure the code runs, but may not produce the right results. Again, very important that this is documented in the README!

Back in the AEA deposit

• Log on to the V1 deposit. You should be in “Modifying” mode.
• Remove (delete) the PII data (my_survey_1.dta)
• Update the README with “PII-laden data for this package can be obtained from ICPSR upon provision of an IRB approval.” in an appropriate location. The ideal README follows the Template README published by the Social Science Data Editors, but make only minimal changes to the README in this deposit.
  • Delete the old README (completely),
  • Upload the new README (in PDF format) to the same location the original one was
• Create a CHANGES.txt. Identify changes made. E.g. “V2: Removed PII-laden data, adjusted code.”. It should look something like this:

V1: Original deposit
V2: Removed PII-laden data, adjusted code.

• Upload the CHANGES.txt.
• (If possible) Link to the DOI of the PII-laden deposit in the “Related publications” section of the deposit (use the Import from DOI functionality, choose is supplemented by for the Relationship field)
• You may have to add the “manuscript number” to the metadata, see this link. Use the last part of the DOI of the paper, e.g. if the paper DOI is https://doi.org/10.1257/aer.p20161120 then the “manuscript number” should be aer.p20161120.
• Re-Submit to the AEA

The end state

After all the fixes,

• V1 is still visible, but can no longer be downloaded.

• V2 is published, contains no PII data, (possibly) links to the deposit of the restricted data, and remains otherwise available to replicators. You are still in compliance with the AEA publication agreements and your obligations as an AEA author!

Final comments

Authors should feel free to reach out to their journal’s data and reproducibility editor with questions about the available options.

Much of the preparation for papers published in Papers and Proceedings is the same as for the AEA journals, see Step 1 for basic preparation & Step 2 if depositing at the AEA Data and Code repository.

However, our compliance check is more limited than the checks conducted for peer-reviewed papers. Your replication package must nevertheless be fully compliant with the AEA Data and Code Availability Policy.

This means: Should a reader or user of your replication package find errors or omissions at a later stage, you will be asked to remedy the situation in line with our Policy on Revisions of Data and Code Deposits.

What do we do?

My replicators will review the deposit, review the README, and send you a filled-out checklist, verifying compliance with each item. If you want to get a perfect score (gamification!) you can find a printable checklist here.

Repository choices

The use of the AEA Data & Code Repository is strongly encouraged. Other repositories and archives considered to be “trusted” may be acceptable - if you ALREADY have a replication package @ Dataverse/Zenodo/etc. please use that (Github is NOT a “trusted” archive).

More information

Check out our full guidance on preparing for P&P.

The Problem

It turns out that online storage solutions, such as Dropbox, OneDrive, and Box (without necessarily limiting ourselves to these), provide a really “convenient” feature: They show you all the thousands of files you have accumulated, including over the 8 years this project has been underway. But they don’t want to put 2TB of data on your laptop. So the files you see are NOT really there… they leave placeholders in place.

What happens when you want to access these files? You double-click on them, and then it takes a bit longer for your Word, Acrobat, or Stata to open the file, because in the background, OneDropBoxDrive is quickly downloading the file for you, as if it had always been there. Sometimes, the application crashes, because OneDropBoxDrive is taking too long, but you try again, and it works. You think “huh, stupid computer” and go on with your work.

So what happens when you zip up these files? Well, it turns out, in many cases (not entirely clear when), ZIP file creators are perfectly happy to NOT be patient, and simply zip up the placeholders. Which are empty files.

When you then upload these ZIP files to deposits like the AEA’s Data and Code Repository, the system there happily unzips the ZIP file. And perfectly preserves a bunch of empty files.

The problem is as bad on other repositories, such as the various Dataverses out there, or Zenodo.

The partial Solution

We have added to our ingest scripts a check for zero-byte files, which will flag these files. But this is relatively new, and not all replicators catch this early on. Leading to frustrating delays for everybody.

The Solution

How do YOU (the author) avoid this problem? Two paths:

• You could simply set the project directory to be “always available offline” (or whatever DriveBoxOneDrop may call it). So they gobble up all the storage space they need. Then, when you ZIP them up, they will all be there. When you are done, you can flip them back to “online only” if you want (until you get back the report from us asking you to fix stuff…)
• Or, you can simply go to the BoxDriveDropOne website, and download the entire project from there, as a ZIP file, ready to upload.

For Dropbox, here’s what you do:

• Go to https://www.dropbox.com/home. If you are not logged on there, go to your Dropbox icon in the system tray (Windows, Linux) or menu bar (Mac), and select “Launch Dropbox Website”.
• Navigate to the folder you want to download. You want to see the files in the folder (be “in” the folder).

• Click on the gear icon ⚙️ to the right of the folder name, and select “Download”

Voilà! You have a ZIP file with all the files in it, ready to upload to your deposit, with no chance at all that some files are missing.¹

The typical table in economics papers will have up to 9 columns, and 5-25 rows. Seeking out, then copy-pasting up to 225 numbers per table is a lot of work, inhibits re-running of code (because “too complicated”), and is a source of errors. There are many packages out there that might allow you to construct a custom table in the programming language of your choice¹, such as (Stata) estout,² outreg2,³ regsave, ⁴, (R) stargazer,⁵, modelsummary,⁶ and many others in R, Julia, Python, etc.

Here, however, I want to emphasize a custom approach that leverages what most people already know: how to use cell references in Excel. I combine a simple export of regression output to Excel sheets, combined with an Excel template. The replication package would contain the (possibly pre-populated) Excel template, and voilà! A reproducible table is born, and updated every time the code is re-run. While you might still need to update the table in your paper (or correctly link an Excel sheet to Word so that it also auto-updates…)

A more complete version, including with downloadable sample code and any improvements and error corrections, is available at https://larsvilhuber.github.io/writing-r-to-excel/.

The Example

The included file regression_results.xlsx has a tab called “My funny table”, which references the content of the two other tabs:

The content of the two tabs Regression1 and Regression2 will be filled with the output from two regressions. I have explicitly not set a seed, so that the values change very time you run the file. You can verify that the “funny table” is automatically updated with the latest results.

You can find the full code as a Quarto (R) file at https://github.com/larsvilhuber/writing-r-to-excel.

Setup

Load required packages and set up environment. The package openxlsx is used in preference to writexl because we explicitly want to preserve the contens of any existing (formatted!) Excel file.

library(tidyverse)

── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
✔ dplyr     1.1.4     ✔ readr     2.1.5
✔ forcats   1.0.0     ✔ stringr   1.5.1
✔ ggplot2   3.5.1     ✔ tibble    3.2.1
✔ lubridate 1.9.4     ✔ tidyr     1.3.1
✔ purrr     1.0.2     
── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
✖ dplyr::filter() masks stats::filter()
✖ dplyr::lag()    masks stats::lag()
ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errors

library(openxlsx)
library(broom)

Data Generation

Create sample datasets for the regression analyses.

# Create sample data for first regression
set.seed(123)
data1 <- tibble(
  x1 = rnorm(100),
  x2 = rnorm(100),
  y1 = 2*x1 + 0.5*x2 + rnorm(100)
)

# Create sample data for second regression
data2 <- tibble(
  z1 = rnorm(100),
  z2 = rnorm(100),
  z3 = rnorm(100),
  y2 = 1.5*z1 - 0.8*z2 + 0.3*z3 + rnorm(100)
)

Analysis

Perform regression analyses on both datasets.

# Run regressions
reg1 <- lm(y1 ~ x1 + x2, data = data1)
reg2 <- lm(y2 ~ z1 + z2 + z3, data = data2)

# Convert regression results to tidy format
reg1_results <- tidy(reg1)
reg2_results <- tidy(reg2)

Save Results

Depending on what you want to do, you could simply post-process the existing tables, creating a simple dataset that is printed. Here, we save the individual results as datasets. If you were to split this document into three parts – Data generation, Analysis, and Tables – then this would be the last part of your analysis part.

# Save regression results as RDS files
saveRDS(reg1_results, "regression1_results.rds")
saveRDS(reg2_results, "regression2_results.rds")

Tables

Formatting Results via Excel

Here, however, we want to simulate what might be a far more complex table.

# Read data back in, as if we were a separate program

reg1_results <- readRDS("regression1_results.rds")
reg2_results <- readRDS("regression2_results.rds")

I use a function here, in order to handle the case when a file exists, and when one doesn’t. The very first time you would use this, no Excel file exists, but subsequent runs would overwrite the tabs within the existing file.

# Function to write to Excel, creating file if it doesn't exist
write_to_excel <- function(results_list, filename) {
  # Check if file exists
  if (!file.exists(filename)) {
    # Create new workbook if file doesn't exist
    message("Creating new file")
    wb <- createWorkbook()
  } else {
    # Load existing workbook if file exists
    message("Re-using existing file")
    wb <- loadWorkbook(filename)
  }
  
  # Loop through the results list and write/overwrite sheets
  for (sheet_name in names(results_list)) {
    # Remove sheet if it already exists
    if (sheet_name %in% names(wb)) {
      removeWorksheet(wb, sheet_name)
    }
    # Add new worksheet
    addWorksheet(wb, sheet_name)
    # Write data to worksheet
    writeData(wb, sheet_name, results_list[[sheet_name]])
  }
  
  # Save workbook
  saveWorkbook(wb, filename, overwrite = TRUE)
}

# Create list of results
regression_results <- list(
  "Regression1" = reg1_results,
  "Regression2" = reg2_results
)

# Write to Excel file
write_to_excel(regression_results, "regression_results.xlsx")

Display Results Here

Show the regression results in the document.

# Display results from first regression

knitr::kable(reg1_results)

# Display results from second regression

knitr::kable(reg2_results)

Directory Contents

Show files created in the working directory.

# List all files in current working directory
list.files(pattern = "(xlsx|rds)$")

[1] "regression_results.xlsx"          "regression_results_template.xlsx"
[3] "regression1_results.rds"          "regression2_results.rds"         

Housekeeping

sessionInfo()

R version 4.4.2 (2024-10-31 ucrt)
Platform: x86_64-w64-mingw32/x64
Running under: Windows 11 x64 (build 26100)

Matrix products: default


locale:
[1] LC_COLLATE=English_United States.utf8 
[2] LC_CTYPE=English_United States.utf8   
[3] LC_MONETARY=English_United States.utf8
[4] LC_NUMERIC=C                          
[5] LC_TIME=English_United States.utf8    

time zone: America/Toronto
tzcode source: internal

attached base packages:
[1] stats     graphics  grDevices datasets  utils     methods   base     

other attached packages:
 [1] broom_1.0.7      openxlsx_4.2.7.1 lubridate_1.9.4  forcats_1.0.0   
 [5] stringr_1.5.1    dplyr_1.1.4      purrr_1.0.2      readr_2.1.5     
 [9] tidyr_1.3.1      tibble_3.2.1     ggplot2_3.5.1    tidyverse_2.0.0 

loaded via a namespace (and not attached):
 [1] gtable_0.3.6      jsonlite_1.8.9    compiler_4.4.2    renv_1.0.11      
 [5] zip_2.3.1         Rcpp_1.0.13-1     tidyselect_1.2.1  scales_1.3.0     
 [9] yaml_2.3.10       fastmap_1.2.0     R6_2.5.1          generics_0.1.3   
[13] knitr_1.49        backports_1.5.0   munsell_0.5.1     pillar_1.9.0     
[17] tzdb_0.4.0        rlang_1.1.4       utf8_1.2.4        stringi_1.8.4    
[21] xfun_0.49         timechange_0.3.0  cli_3.6.3         withr_3.0.2      
[25] magrittr_2.0.3    digest_0.6.37     grid_4.4.2        rstudioapi_0.17.1
[29] hms_1.1.3         lifecycle_1.0.4   vctrs_0.6.5       evaluate_1.0.1   
[33] glue_1.8.0        fansi_1.0.6       colorspace_2.1-1  rmarkdown_2.29   
[37] tools_4.4.2       pkgconfig_2.0.3   htmltools_0.5.8.1

So what happens when an author inadvertently does share such data, and the data provider finds out, and is not happy? Let’s call this a “take-down request.” The data provider asks that the author remove the data from the replication package. But the author is committed (by signing an agreement) to not withdrawing the replication package. How to proceed? Read on!

In a nutshell

The gist of the solution is the following, for AEA manuscripts (which I obviously know well), but probably similarly for other journals:

• Remove the infringing data from V1 of the replication package.
• [Ideally] Create a new, separate deposit that is in compliance with the data provider’s terms of use.
• Document the new location of the now absent data in the README of the replication package.
• Submit the revised replication package to the journal.
• The journal publishes V2 of the replication package, which is now compliant.

An example: PSID data

For years, I as Data Editor have alerted authors when I detected data that infringes on terms of use. Two of the most frequent cases are the Panel Study of Income Dynamics (PSID) and the UNSD’s Comtrade data. We even have an FAQ entry dedicated to the PSID case and a copy-and-paste entry in our “most frequent errors” document.

But what if PSID data does end up in a deposit, either because I missed it (believe it or not, I’m not perfect), or because the replication package was published before the Data Editor was appointed?¹ In fact, on November 1, 2024, the folks at PSID did just that: Contacted a large number of authors that had included PSID data in their replication packages, asking them to remove the data.

It is worth reminding that the PSID terms of use, which everybody agrees to when downloading the data, are quite clear:

1. Not transfer PSID public data that has been downloaded from the website, including user-created data extracts, to any third parties…

So what can authors do?

Preliminary steps

The first steps involve getting access to your own replication package. This may differ by when the replication package was published.²

• For authors with papers published before 2019 (approximately)
  • All authors will need to get an openICPSR login, which will serve them both for the changes to the AEA and for creating the new PSID deposit.
  • They will need to request access to the deposit. The openICPSR helpdesk may be able to help with that. The authors should mention the URL or DOI, and the openICPSR login they want the deposit associated with.

• For authors with more recent papers:

  • They will need to re-create an ICPSR account with the same email as previously used since ICPSR changed their login system in Feb 2024. The openICPSR helpdesk may help with that.
  • They need to then create a new version.

]

Rules that apply to revisions of replication packages

Here are a few simple rules:

• The modified package should differ only as necessary from the original package. Less work for the author, greater clarity for replicators, everybody wins.
• Journal-specific policies may apply. For instance, the AEA has a Policy on Revisions of Data and Code Deposits in the AEA Data and Code Repository. Among other things, it requires that all changes be clearly identified.

Fixing the problem

Once the preliminaries are done, here’s what authors need to do to remain compliant with AEA policy (similar rules may apply at other journals):

• PSID actually has a way to properly and openly preserve the user-created extracts in compliance with their terms of use. It is the PSID Repository on openICPSR.
• For sake of discussion, the “original replication package” is V1. The replacement replication package will be V2.

• For sake of discussion, consider the case where V1 contains a single file psid.dta.

PSID deposit

• Authors should create a new deposit at the PSID repository with title “Supplementary data for: [NAME OF THE MANUSCRIPT]”.
  • Put into it ONLY the PSID data psid.dta. Do not put other things into this deposit.
  • You might want to add the DOI of V1 and the DOI of the manuscript in the “Related publications” section (use the Import from DOI functionality, choose is supplemented by for the Relationship field)
  • Publish it.
  • Remember the DOI (not the URL). Call this the “[PSID DOI]”. Replace this with the actual DOI everywhere it is mentioned below!

Back in the AEA deposit

• Log on to the V1 deposit. You should be in “Modifying” mode.
• Remove (delete) the same PSID data (psid.dta) you just uploaded into the PSID deposit, from the openICPSR AEA deposit V1.
• Update the README with “PSID data for this package can be obtained from [PSID DOI]” in an appropriate location. The ideal README follows the Template README published by the Social Science Data Editors, but make only minimal changes to the README in this deposit.
  • Delete the old README (completely),
  • Upload the new README (in PDF format) to the same location the original one was
• Create a CHANGES.txt. Identify changes made. E.g. “V2: Moved PSID data to [PSID DOI]”. It should look something like this:

V1: Original deposit
V2: PSID data moved to [PSID DOI]

• Upload the CHANGES.txt.
• Link to the [PSID DOI] in the “Related publications” section of the deposit (use the Import from DOI functionality, choose is supplemented by for the Relationship field)
• You may have to add the “manuscript number” to the metadata, see this link. Use the last part of the DOI of the paper, e.g. if the paper DOI is https://doi.org/10.1257/aer.p20161120 then the “manuscript number” should be aer.p20161120.
• Re-Submit to the AEA
• Contact the openICPSR helpdesk and ask them to “Unpublish” the V1 deposit.

The end state

After all the fixes,

• V1 is still visible, but can no longer be downloaded.

• the PSID deposit is now available. Users who want to download the data have to agree to the PSID’s terms of use. You are now in compliance with the PSID’s terms of use!
• V2 is published, contains no PSID data, links to the PSID deposit, and remains otherwise available to replicators. You are still in compliance with the AEA publication agreements and your obligations as an AEA author!

An example

Fran Blau and Larry Kahn had a JEL article (with replication package!) back in 2016:

Blau, Francine D., and Lawrence M. Kahn. “The Gender Wage Gap: Extent, Trends, and Explanations.” Journal of Economic Literature 55, no. 3 (September 2017): 789–865. https://doi.org/10.1257/jel.20160995.

The relevant original replication package was migrated to openICPSR in 2019, and its contents can still be viewed here, but not downloaded:

Blau, Francine D., and Kahn, Lawrence M. Replication data for: The Gender Wage Gap: Extent, Trends, and Explanations. Nashville, TN: American Economic Association [publisher], 2017. Ann Arbor, MI: Inter-university Consortium for Political and Social Research [distributor], 2019-10-12. https://doi.org/10.3886/E113913V1

The amended replication package is now available, and linked from the earlier one (note the V2, and the later publication date):

Blau, Francine D., and Kahn, Lawrence M. Replication data for: The Gender Wage Gap: Extent, Trends, and Explanations. Nashville, TN: American Economic Association [publisher], 2024. Ann Arbor, MI: Inter-university Consortium for Political and Social Research [distributor], 2024-11-14. https://doi.org/10.3886/E113913V2

The required CHANGES.txt reads

V1: Original version
V2: Moved PSID data to https://doi.org/10.3886/E210483V1 and updated README.

and the new PSID deposit is referenced there, in the related publications, and in the README.

Kahn, Lawrence, and Francine Blau. “Supplementary for: Francine D. Blau and Lawrence M. Kahn, ‘The Gender Wage Gap: Extent, Trends, and Explanations’, Journal of Economic Literature , 55 ,3 (Sept 2017): 789-865.” ICPSR - Interuniversity Consortium for Political and Social Research, 2024. https://doi.org/10.3886/E210483V1.³

Conclusion

The problem outlined above is not unique to the PSID. We have seen “takedown” requests from the German Socio-Economic Panel and others, and I have prevented inadvertent publication of UNSD Comtrade data, Compustat data, commercial data, PII-laden data, etc. The PSID is relatively unique in that it has a mechanism to actually preserve the author-created extracts. A few other such mechanisms exist, but they are by no means universal. Most “takedown” requests can only be satisfied with the deletion of data, and the inclusion of detailed information on how others can obtain the data from the original data provider - an error-prone and not ideal process.

Authors should feel free to reach out to their journal’s data and reproducibility editor with questions about the available options.

It introduces lots of failure points - not necessarily in the final package, but in the development process.

The example

Consider that you are halfway through your project. You have a base model, and three variants thereof. Each of which is the base model, copied into three different directories, with minimal modifications in the code. For sake of clarity, and with no loss of generality intended:

code/
    base-model/
            base-model.m
    robustness-check-1/
            base-model.m
    robustness-check-2/
            base-model.m
    robustness-check-3/
            base-model.m

where a diff between base-model.m and robustness-check-1/base-model.m would show only a few lines changed:

diff code/base-model/base-model.m code/robustness-check-1/base-model.m
13073c13073
< a=1;
---
> a=2;

Note: the change would appear to be in line 13073, which is awfully subtle… this is inspired by real-life examples!

The problem

Now imagine that your thesis advisor, or a reviewer, identifies a key problem in the implemented algorithm. You fix it in base-model.m, but now need to fix it in all the other modified copies too. Will there be any typos when you implement the patch? What if there are not three robustness checks, but 30? Or 300 different parameterizations of a simulation?

The techy programmer might decide to do this with patch in conjunction with git, but that already counts as advanced functionality (I probably lost a lot of readers with the first part of that sentence…)

# create a patch from the updates to the base model
git diff d362f6..3bd22 > patch.diff
# apply the patch to the robustness check
patch code/robustness-check-1/base-model.m < patch.diff

but that is not a robust solution that scales to the 30 or 300 robustness checks.

A possible solution

The core solution is to apply elementary best programming practices: write functions.¹

So should one go about the above problem? My rule of thumb is:

• Copy code once, fine, don’t do it again.
• Copy it twice, stop, and create a function.
• Copy it three times: don’t even go there!

Assuming that you are using MATLAB (where we seem to see this often), you should treat the core model code as a “function” (even if you do not explicitly program it that way), and then call that “function” multiple times. In a nutshell:

code/
    core-simulation/
            parametrized-simulation.m
    base-model.m
    robustness-check-1.m
    robustness-check-2.m
    robustness-check-3.m

where

% base-model.m

% We set the base model to have the following parameters,
% as per Section 2 of the paper:
a=1;
b=2;
c=3;

% We then call the core simulation code
parametrized-simulation

will call the code in “parametrized-simulation.m” with parameters a,b,c already set. The simulation itself assumes those parameters are set.²

The robustness checks might then have

% robustness-check-1.m

% We set the model to have the following parameters,
% as per Section 3 of the paper:

a=2; % This differs from the base model
b=2;
c=3;

% We then call the core simulation code
parametrized-simulation

or go through a loop of multiple values:

% robustness-check-2.m


% We set the model to have the following parameters,
% as per Section 4 of the paper:

b=3;
c=4;

% We now explore the space of a within this context.
for a = 1:10
    parametrized-simulation
end

% Summarize results from exploration

summarize-results

Other languages

Is that a MATLAB-specific problem? No. We see similar issues in Stata, R, Python, and other languages. The core solution is the same: write functions, and call them multiple times, instead of copying code around, and hard-coding parameters.

Other situations

The above example had continuous (well, kind of) parameter spaces. What if they are discontinuous? You can still loop over a whole set of specifications by storing them in a file - yes, an Excel file is fine! - and reading them in (in MATLAB, readtable). For example,

% Read the Excel spreadsheet
[num, txt, raw] = xlsread('your_spreadsheet.xlsx');

% Get the number of rows in the data
[numRows, ~] = size(raw);

% Loop over each row, starting from the second row (assuming first row is header)
for i = 2:numRows
    % Extract values from columns A, B, and C
    a = raw{i, 1};  % Column A
    b = raw{i, 2};  % Column B
    c = raw{i, 3};  % Column C
    
    % Call the function something() with the extracted values
    result = something(a, b, c);
    
    % You can do something with the result here if needed
    % For example, store it in an array or print it
    disp(['Result for row ', num2str(i), ': ', num2str(result)]);
end

(straight from Claude AI)

Summarizing it and creating tables

Of course, you now also have to properly capture all those results, store them, and be able to summarize them, and create tables from them. That’s another story.

A scenario

You are happily browsing along the internet, and find an fabulous dataset you think you can use for your awesome idea. Download, analyze, submit, publish, … all good. But then the pesky journal asks for a replication package. Easy, right? Wrap up the code, the data, upload, … all good? Maybe. Possibly not.

The problem

Many data used by economists stems from other data sources. Most of these data sources are on the internet. Anybody can use data that’s on the internet and do with it what they want, right?

As the discussion surrounding copyright and artificial intelligence shows, the answer is complex.¹² There is a concept of “fair use” in US copyright law, and in particular, it has been used to allow for legal webscraping.³

But that applies primarily for “use”. If you want to publish the data you obtained from elsewhere, that is no longer necessarily considered (fair) “use”, it falls under “redistribution” or “publication”.

The solution

In principle, the solution is for data providers to establish clear terms of use and licenses. And many do, especially in the commercial space. In general, you will find that “terms of use” restrict certains uses and rights, and “licenses” allow or grant certain uses and rights.

Historically, it was software that was licensed, and “open source” licenses have a long history. Many such licenses are listed at the “open source initiative”.

For data, a popular set of licenses are the “Creative Commons” licenses. These licenses are designed to be easy to understand and to use, and allow for a wide range of uses - including redistribution. The CC-BY license is abundantly clear about what it allows:

CC By 4.0 DEED:

You are free to:

• Share — copy and redistribute the material in any medium or format for any purpose, even commercially. […]

Other data licenses exist that allow for redistribution explicitly:

• Open Data Commons Open Database License (ODbL)
• various government “open data” licenses, such as the UK Open Government License and Statistics Canada’s Statistics Canada Open Licence.

(often times, the differences are minor between these licenses).

The problem, part 2

The problem is that many (academic) data providers still do not show or apply these licenses, and have no terms of use, or ones that are hidden. It thus becomes difficult for researchers to know what they can and cannot do with the data they find, leading to avoidable confusion.

Sometimes, these are identified at the time of downloading the data. For instance, the widely-used World Values Survey displays four conditions to researchers who (after registering) download the data:

It otherwise mentions the “non-redistribution license” or that “data redistribution is prohibited” in multiple places, but there is no single “Terms of Use” page, or a “WVS License” page that is easily accessible and referenceable.

Even when terms of use are provided, permissions and restrictions are not necessarily linked together. For instance, the fabulous DHS Program states on many of its pages

The DHS Program is authorized to distribute, at no cost, unrestricted survey data files for legitimate academic research.

but one has to search for the Terms of Use, tucked in between “Contraceptive Calendar Tutorial” and “DHS Github Code Share”, which state (also very clearly)

Conditions of Use for The DHS Program datasets (applies to all datasets downloaded from The DHS Program website: www.dhsprogram.com)

[…]

• Agree that the datasets will not be shared with other researchers without the written consent of The DHS Program.

(emphasis added). Many researchers include DHS data in their replication packages, and active data editors like myself have to remind them to remove the data from the replication package.

The takeaways

Data providers

• Data providers should have a clear “Terms of Use” page, or a “License” page, that is easily accessible and referenceable.
• They should also include the license and conditions in every download.
• When data formats that allow for annotation (Stata, SAS), embed the license information in the dataset itself.

Researchers

• Always record the terms of use and license when you download the data.
• Search for “terms of use” or “license” on data providers’ websites.
• If the website asks for registration, it is highly likely that there are redistribution restrictions.
• Request that data providers make their terms of use and licenses more visible and accessible.
• Consult (and update) the Social Science Data Editors’ Terms of Use database.