Description¶
Converts the manuscript PDF(s) in a directory to Markdown and extracts the figures as images. The conversion is mechanical: nothing is interpreted, summarized or corrected. The m-parse-manuscript pipeline runs it through 31_parse_manuscript.sh and commits the result as manuscript_parsed/.
A file is taken to be the manuscript when its name is
PDF_Proof.PDF(any case), ora journal acronym followed by the manuscript number, such as
AEJApp-2025-0453.R1_Proof_hi.pdforAER-2025-0473_AO.pdf.
Other PDFs (forms, Data and Code Availability.pdf, 13071-2.pdf) are ignored.
Usage¶
python3 tools/manuscript_to_markdown.py [--source-dir DIR] [--output-dir DIR] [--engine auto|marker|markitdown] [--list]
python3 tools/manuscript_to_markdown.py [--source-dir DIR] [--checksum-file FILE] --needs-conversion
python3 tools/manuscript_to_markdown.py [--source-dir DIR] [--checksum-file FILE] --record-checksums--source-dir- Directory with the PDFs (default: current directory).--output-dir- Where the results go (default:manuscript_parsed). Each manuscript gets<output-dir>/<file stem>/, holding<file stem>.mdand the extracted images.--engine-markerrunsmarker_singlewith OCR disabled (layout, tables, equations, figures as images).markitdownextracts text only: fast, no images.auto(default) tries marker and falls back to markitdown if marker is missing or fails.--list- Print the files that would be converted and stop.--checksum-file- SHA-256 checksums of the manuscripts last converted, insha256sumformat (default:generated/manuscript.sha256).--needs-conversion- Compare the manuscripts with the checksum file and stop, without converting. Exit code 0 if they need converting: the file is missing or empty, or a manuscript was changed, renamed, added or removed. Exit code 3 if they are unchanged or there are none. Other PDFs do not count.--record-checksums- Write the checksums of the manuscripts to the checksum file and stop, without converting. Writes nothing if there are none.
Without these two options: exit code 0 on success or when no manuscript is found; 1 if a manuscript could not be converted by any engine. A new run replaces the earlier result for the same file.
Which converter¶
Measured on two 90-page proofs (PDF_Proof.PDF and an AEJ: Applied proof), without OCR:
| markitdown 0.1.9 | marker 1.10.2 | marker 2.0.0 | |
|---|---|---|---|
| Time, CPU, 8 threads | 4-47 s | 16-33 min | 30-34 s |
| Time, GPU (RTX 3060) | - | 1-2 min | 27-30 s |
| Peak memory (CPU) | 0.1-1.7 GB | 12.5 GB | 2.3 GB |
| Figures as images | none | 9 and 31 | 7 and 25 |
| Equations | broken into table fragments | LaTeX | flattened text |
| Words run together | yes (AEJ proof) | no | no |
The figures in these PDFs are vector graphics, so only marker’s layout detection recovers them. Marker 2.0 gives byte-identical output on CPU and GPU, so a GPU buys nothing; the Bitbucket-hosted workers have none.
The default is marker 2.0: it is about 50 times faster and needs a fifth of the memory. Compared with 1.10.2 it finds the figures on the same pages, with these differences:
A multi-panel figure becomes one image where 1.10.2 gives one image per panel (hence 25 instead of 31 images); no panel is lost.
The journal logo on the title page is not extracted.
Equations come out as flattened text instead of LaTeX.
Two figure titles (Figures 2 and 3 of the PDF_Proof sample) are not in the text; they are probably part of the image.
See requirements-manuscript.txt to switch to 1.10.2.
Requirements¶
pip install -r requirements-manuscript.txt (marker and markitdown). On a machine without a GPU, install the CPU build of torch first (see the comment in that file).