Skip to article frontmatterSkip to article content
Site not loading correctly?

This may be due to an incorrect BASE_URL configuration. See the MyST Documentation for reference.

manuscript_to_markdown.py - Markdown version of the manuscript

Description

Converts the manuscript PDF(s) in a directory to Markdown and extracts the figures as images. The conversion is mechanical: nothing is interpreted, summarized or corrected. The m-parse-manuscript pipeline runs it through 31_parse_manuscript.sh and commits the result as manuscript_parsed/.

A file is taken to be the manuscript when its name is

Other PDFs (forms, Data and Code Availability.pdf, 13071-2.pdf) are ignored.

Usage

python3 tools/manuscript_to_markdown.py [--source-dir DIR] [--output-dir DIR] [--engine auto|marker|markitdown] [--list]
python3 tools/manuscript_to_markdown.py [--source-dir DIR] [--checksum-file FILE] --needs-conversion
python3 tools/manuscript_to_markdown.py [--source-dir DIR] [--checksum-file FILE] --record-checksums

Without these two options: exit code 0 on success or when no manuscript is found; 1 if a manuscript could not be converted by any engine. A new run replaces the earlier result for the same file.

Which converter

Measured on two 90-page proofs (PDF_Proof.PDF and an AEJ: Applied proof), without OCR:

markitdown 0.1.9marker 1.10.2marker 2.0.0
Time, CPU, 8 threads4-47 s16-33 min30-34 s
Time, GPU (RTX 3060)-1-2 min27-30 s
Peak memory (CPU)0.1-1.7 GB12.5 GB2.3 GB
Figures as imagesnone9 and 317 and 25
Equationsbroken into table fragmentsLaTeXflattened text
Words run togetheryes (AEJ proof)nono

The figures in these PDFs are vector graphics, so only marker’s layout detection recovers them. Marker 2.0 gives byte-identical output on CPU and GPU, so a GPU buys nothing; the Bitbucket-hosted workers have none.

The default is marker 2.0: it is about 50 times faster and needs a fifth of the memory. Compared with 1.10.2 it finds the figures on the same pages, with these differences:

See requirements-manuscript.txt to switch to 1.10.2.

Requirements

pip install -r requirements-manuscript.txt (marker and markitdown). On a machine without a GPU, install the CPU build of torch first (see the comment in that file).