An OCR system can read most of a page correctly and still get the one value your workflow depends on wrong. A misplaced decimal, a missing checkbox or a number taken from the wrong row can matter more than dozens of correctly recognized words.
To test OCR accuracy on scanned documents, compare outputs with independently checked expected values, score the fields your application uses, and repeat the evaluation across realistic input conditions. Keep text recognition, field extraction and the downstream workflow decision distinguishable so you can locate failures.
1. Define what a correct result means
Start with one document type and one business task. For a fictional tax form, that might mean extracting the tax year, employee name, wages and tax withheld. For a laboratory requisition, it might mean identifying the requested test and matching the requisition to a specimen label.
Write the scoring rules before running the system. Decide which fields are required, whether a field can repeat, and how to handle blank, unreadable or ambiguous content.
Normalize only differences your application considers equivalent. A monetary amount written as 98,500.00 may be equivalent to 98500 when the currency is already established. Removing leading zeros from an identifier may change its meaning. A value found on the wrong page or assigned to the wrong field should not pass simply because it appears somewhere in the document.
2. Build ground truth you can trust
Ground truth is the reference against which output is checked. Record the expected field, value and source location, along with any relevant relationship to another document. Have the reference reviewed independently of the system being evaluated.
For generated documents, retain the configured values and verify that the rendered artifact actually displays them as intended. A generation record is not enough if clipping, overlap or a template error makes the value different or invisible on the page.
Distinguish a source value that exists from a value that can be recovered from a particular image. If severe degradation makes a field unreadable, a useful expected behavior may be to request review rather than guess. Preserve the underlying value for analysis, but score escalation separately from extraction correctness.
Keep development and evaluation data separate. Variants derived from the same original document should stay in the same split when measuring generalization. Otherwise, near-duplicate layouts or content can make results look stronger than they are.
3. Vary the conditions while keeping the answers controlled
Begin with a readable reference document. Create controlled variations that reflect how content reaches your organization:
- Low contrast and uneven lighting: Faint text, shadows and photographed pages.
- Blur and compression: Lost character detail or degraded image quality.
- Fax and photocopy artifacts: Broken strokes, noise and repeated reproduction.
- Skew and rotation: Changes to page alignment and reading order.
- Handwriting and corrections: Mixed typed and handwritten content, annotations and overwritten fields.
- Incomplete or conflicting inputs: Missing pages, blank fields and inconsistent values across related documents.
Test one condition at a time first, then combinations that occur in the real workflow. Record the severity and how each variation was produced. Ten copies of one blurred form are not ten independent document layouts.
Tesseract's quality guidance describes how noise, skew and image processing can affect recognition. The practical implication is to record preprocessing as part of the configuration and evaluate it, rather than silently changing inputs between runs.
GritWorks' OCR test examples use one fictional W-2 under low-light, blur, fax and photocopy conditions, with reference values. They are a starting point for understanding the method, not a production benchmark or evidence of product accuracy.
4. Score the fields, then the document
For a fixed schema with one expected value per required field, report required-field match rate: correctly returned required values divided by all expected required values. Missing values count as failures. Report each critical field separately as well as the aggregate.
For optional or repeated entities, also track false additions and omissions. Precision measures how many returned entities match the reference; recall measures how many reference entities were returned correctly. F1 combines those measures. Define matching and duplicate-handling rules before comparing systems. Google Cloud's Document AI evaluation guidance provides a useful reference for this approach.
Then calculate a document pass rate using your stated acceptance rule. For example, a document may pass only when every critical field is correct and required cross-document checks succeed. Report invalid output, timeouts and processing failures rather than dropping them from the denominator.
A worked example: 97% can hide a critical weakness
Imagine 20 fictional documents with 10 required fields each. The system returns 194 of the 200 expected field values correctly. Its aggregate required-field match rate is 97%.
Now suppose all six errors occur in the amount field, on six different documents. The amount field is correct on only 14 of 20 documents: 70%. If every critical field must be correct, only 14 documents pass, also 70%.
These numbers are illustrative, not measured GritWorks results. They show why an overall score cannot replace per-field analysis. A result that is useful for routing a document to a reviewer may be unsuitable for a workflow that acts on the amount without review.
5. Separate extraction quality from review behavior
A confidence score is not proof that a value is correct. Evaluate the threshold and escalation policy against labeled examples. If you raise a threshold, report both the quality of accepted outputs and the proportion sent for review.
Track at least these outcomes: correct automatic output, incorrect automatic output, appropriate escalation, and unnecessary escalation. A system that sends everything for review should not appear equivalent to one that produces dependable automatic results.
Choose thresholds on development or validation data, then assess the chosen policy on a separate evaluation set. Do not tune repeatedly on the final test set and describe the result as independent validation.
6. Keep a scorecard for every run
Copy the following fields into your evaluation record:
- Run identity: Date, model or processor version, extraction schema, prompt where applicable, preprocessing settings and scoring-rule version.
- Dataset: Version, document count, source-document families, required fields and known-answer reference.
- Coverage: Document types, layouts, handwriting styles, degradation conditions and severity.
- Field results: Matches, missing values and incorrect values for each required field; false additions and entity precision/recall where applicable.
- Workflow results: Document pass rate, processing failures, review rate and incorrect outputs accepted without review.
- Operations: Processing latency, review effort and test environment, measured under comparable conditions.
- Decision: Acceptance criteria, failed cases, responsible owner and the change required before the next run.
Show sample counts beside percentages. A small demonstration set can reveal a failure but cannot establish how often it occurs in production. Keep a representative evaluation set separate from a deliberately difficult stress set, and report both.
7. Turn failures into regression tests
Retain the inputs, expected results and configuration for failures you investigate. Run the same cases after changes to the model, prompt, schema or preprocessing. Where outputs can vary, repeat runs under documented settings and report that variation.
Keep new holdout cases for checking generalization as well as the regression cases used during development. A fix that repairs a known example still needs evidence that it works on other documents.
Set release criteria around the operational consequence of errors. There is no single OCR accuracy percentage that makes every banking, insurance or healthcare workflow ready for production.
Where GritWorks fits
GritRender creates controlled synthetic document variations with known values. GritTransform extracts structured JSON from PDFs and images, which can be compared with those values. You can also evaluate an existing OCR or extraction system against the same inputs. GritRedact prepares reviewed derivatives when approved real documents contain sensitive information that must be removed.
All three products are available offline on-premises and through sales-assisted SaaS. GritWorks' laboratory-requisition case study describes a technical evaluation of controlled variations; it is not a universal accuracy claim.
Start with a document type, the fields your application needs and the failure conditions you want to test. Bring those requirements to a working session to define a repeatable evaluation. Use fictional examples initially and agree a controlled route before sharing sensitive enterprise content.
References and examples
Google Cloud — Document AI evaluation: https://docs.cloud.google.com/document-ai/docs/evaluate
Tesseract — Improving output quality: https://tesseract-ocr.github.io/tessdoc/ImproveQuality.html
GritWorks — OCR examples and reference values: https://gritworks.ai/resources/ocr-test-examples
GritWorks — Laboratory-requisition technical evaluation: https://gritworks.ai/case-studies/laboratory-ocr-testing
