← Free datasets and benchmarks

Docling OCR on degraded scans: a reproducible benchmark on matched clean and degraded pages

The same synthetic page, scanned clean and then blurred, skewed, photocopied, shot in low light or faxed. We ran Docling's CPU OCR engines over every version and scored each populated field.

THE SHORT ANSWER

On 14 Latin-script synthetic pages (291 fields per condition), RapidOCR matched 98.6% of fields on clean scans and 68.4% on fax; ocrmac 92.8% and 61.5%; EasyOCR 70.5% and 17.2%. Fax was the hardest synthetic condition for every engine. This is a first pass on synthetic documents with simulated degradations, run on CPU.

Independent benchmark. GritWorks published this benchmark and supplied the synthetic documents. It is not affiliated with, reviewed by or endorsed by IBM or the Docling project. The numbers describe open-source OCR engines inside Docling on one synthetic set; they are not a ranking of tools in general.

Code: github.com/darshilbabel/docling-degraded-scan-bench (Apache-2.0). Data: huggingface.co/datasets/Gritworks/docling-degraded-scan-bench (CC BY 4.0).

Results: field exact match by engine and condition

Common subset: 14 Latin-script pages with 291 scored fields per condition, identical images for every engine. Each cell is field exact match (EM, %, higher is better), with field CER (%, lower is better) in brackets.

Field exact match and CER by OCR engine and condition, Latin-script pages
EngineCleanBlurSkewPhotocopyLow lightFax
ocrmac (Apple Vision)92.8 (3.4)85.6 (3.2)92.1 (5.2)91.4 (1.1)95.9 (0.7)61.5 (7.6)
RapidOCR98.6 (0.2)93.1 (1.2)97.6 (0.3)91.4 (1.1)97.2 (0.5)68.4 (9.5)
EasyOCR70.5 (7.3)4.8 (89.6)62.2 (9.8)57.0 (12.5)63.6 (10.3)17.2 (52.6)
  • Fax was the hardest synthetic condition for every engine. RapidOCR and ocrmac kept field CER under 10% on fax, so most misses were near-misses: a lost slash, a broken digit or a merged word.
  • Skew and low light stayed within a few points of the clean score for RapidOCR and ocrmac.
  • Photocopy mostly cost punctuation and spacing. On the full 27 Latin pages, RapidOCR scored 90.9% strict EM and 97.9% when spaces and punctuation are ignored.
  • Blur cost RapidOCR and ocrmac 5 to 7 points. EasyOCR fell to 4.8% under Docling's default settings (see the confidence-threshold note under Method).
  • On the 38 degraded pages shipped with the sample packs, the packs' strong blur took all three engines to 0% EM.
  • On the full 27 Latin pages, ocrmac and RapidOCR show the same ordering of conditions: fax lowest for both.

Hindi, Marathi and Bengali pages

Field EM (%) on the Devanagari (Hindi, Marathi) and Bengali pages. "n/a" means the engine has no recognition model for that script in this setup: Apple Vision has no Indic scripts, and RapidOCR's ONNX models have no Bengali.

Field exact match by OCR engine and condition, Indic pages
Engine · scriptCleanBlurSkewPhotocopyLow lightFax
RapidOCR · Devanagari (Hindi, Marathi)44.916.336.726.538.812.2
EasyOCR · Devanagari (Hindi, Marathi)42.98.232.620.430.62.0
EasyOCR · Bengali41.90.054.832.335.53.2
RapidOCR · Bengalin/an/an/an/an/an/a
ocrmac (Apple Vision) · Devanagari and Bengalin/an/an/an/an/an/a

No CPU engine in this run reached 60% field EM on the clean Hindi, Marathi or Bengali forms, and on fax every engine scored 12.2% or lower. The forms come from the Hindi, Marathi and Bengali OCR test documents pack.

Method

  • Pages. 37 source pages from 30 clean documents in the six public GritWorks sample packs: 27 Latin-script, 6 Devanagari (Hindi, Marathi) and 4 Bengali. Families include invoices and purchase orders, bank statements, a paystub, a W-2, utility bills, a CMS-1500 claim, an explanation of benefits, lab forms, passport and ID specimens, and Indian account-opening forms.
  • Matched pairs. From each clean page, five degraded versions were generated with a fixed seed: blur (Gaussian, σ ≈ 2 px at 150 DPI), skew (2–5° rotation), photocopy (harsh contrast, uneven toner, specks, heavy JPEG), low light (gamma, vignette, sensor noise, warm cast) and fax (half vertical resolution, binarisation, scan-line streaks). Skew boxes go through the same affine transform as the image, so the ground truth carries over exactly. With the 38 pack-native degraded pages, that makes 260 page images at 150 DPI.
  • Ground truth. 766 scored fields per condition: the value exactly as printed (currency signs included) and its bounding box. 45 checkbox and one-character fields are kept but not scored. Ground truth ships as GritWorks JSON, DoclingDocument and DocLang; all 260 DocLang files pass the reference validator.
  • Engines and settings. Docling 2.150.1 on CPU with full-page OCR, table structure off and OCR scale 1.0 (the scan's native 150 DPI rather than Docling's default 3× upsample). EasyOCR ran on a seeded, script-stratified subset of 19 source pages; the engine tables compare only pages every engine ran.
  • Metrics. Field EM: the normalised printed value appears in the OCR lines inside the field's box, padded by half its height. Field CER and WER: edit distance to the best-matching span in that box text, divided by value length. Drift CER: how much an engine's page text changes between the clean and degraded version of the same page.
  • Ablations (4 pages). Docling's default OCR scale of 3.0 changed EM by a few points at most and doubled EasyOCR's time. Lowering EasyOCR's confidence threshold from 0.5 to 0.0 cut blur CER from 81% to 28% and fax CER from 40% to 11%, so EasyOCR's low numbers partly reflect a default filter, not only recognition.

Limitations

  • Synthetic data. Template-generated documents with clean typography and consistent layouts. Real archives vary more in fonts, handwriting, stamps and layout.
  • Simulated degradations. One severity band each. They approximate, but do not reproduce, real scanners, fax machines and phone cameras.
  • First pass, small sample. 14 to 37 pages per comparison, and EasyOCR ran on 19 source pages. Differences of a few points are within noise; there are no confidence intervals yet.
  • CPU-only engines. RapidOCR, EasyOCR and ocrmac on a laptop CPU. Tesseract, GPU engines and the VLM presets were not run, and speeds were measured with engines running side by side, so they are an ordering only.
  • Field-level ground truth. Static labels and boilerplate are not annotated, so full-page CER and reading order are not scored. The matching rule (a substring inside a padded box) is fairly lenient and still under discussion.

Reproduce it

The pipeline downloads the six public sample packs, verifies their SHA-256, rebuilds the matched pairs and runs every engine. run_eval.py --aggregate-only rescores the saved predictions from Hugging Face without running OCR, so you can try your own matching rules.

git clone https://github.com/darshilbabel/docling-degraded-scan-bench
cd docling-degraded-scan-bench
python3.12 -m venv .venv
.venv/bin/pip install -r requirements.txt
.venv/bin/python -I scripts/fetch_packs.py   # download the 6 public packs, verify SHA-256
.venv/bin/python -I scripts/inventory.py
.venv/bin/python -I scripts/build_pairs.py   # deterministic: byte-identical images
.venv/bin/python -I scripts/to_doclang.py
.venv/bin/python -I scripts/run_eval.py      # results/*.json, results/SUMMARY.md

The dataset on Hugging Face holds the page images, ground truth and Docling's saved predictions. Methodology questions and issues are welcome on GitHub.

Common questions

What does this benchmark measure?

How Docling's OCR output changes when the same page is blurred, skewed, photocopied, shot in low light or faxed. Each populated field is scored by exact match: the printed value must appear in the OCR text found inside the field's padded box. Field CER, WER and drift CER are reported alongside.

Which Docling OCR engines were tested?

RapidOCR (Docling's default, on onnxruntime), EasyOCR (PyTorch, CPU) and ocrmac (Apple Vision), with Docling 2.150.1 on CPU. Tesseract was not installed on the run machine, and GPU-class engines and VLM presets were left for a separate run.

Why are EasyOCR's numbers low?

Partly because of a default setting. Docling's EasyOCR integration drops text below 0.5 confidence. In a four-page ablation, setting the threshold to 0.0 cut EasyOCR's blur CER from 81% to 28% and its fax CER from 40% to 11%. EasyOCR also ran on a 19-page subset because of its CPU time.

Are the documents real?

No. All 37 source pages are synthetic, generated from templates, with fictional people, organisations, identifiers and amounts. ID and passport pages are specimens. The degradations are image-processing simulations, not real scanners, fax machines or phone cameras.

Can I use the dataset and code?

Yes. The data and published tables are licensed CC BY 4.0, with attribution to GritWorks, and the code is Apache-2.0. Docling, EasyOCR, RapidOCR, ocrmac and the other dependencies keep their own licences.

Is this benchmark endorsed by IBM or the Docling project?

No. It is an independent benchmark published by GritWorks, which supplied the synthetic documents. It is not affiliated with, reviewed by or endorsed by IBM or the Docling project, and it does not rank vendors or tools in general.