A document-extraction release looks better than the previous one. But the test folder changed, two expected values were corrected, and the normalization script now strips punctuation. How much of the improvement belongs to the release?
A regression manifest makes that question answerable. It is a versioned record connecting each test input to its expected result, the system configuration used, the scoring rules and the review decision. It does not need to be a new platform. A small team can start with a reviewed CSV or JSON file and an access-controlled folder of artifacts.
The goal is not to collect more metadata. It is to make a failed test reproducible and a release comparison explainable.
Separate the test case from the test run
Keep two records. A test case describes what should be tested: document identity, scenario, expected fields and intended workflow behavior. A test run records what happened when a particular system processed that case.
Do not overwrite the case with the latest output. Otherwise, the expected value can quietly become whatever the current system produced. Keep model output, reference data and human corrections distinguishable.
For a fictional invoice case, the case record might identify an invoice PDF, the expected currency and total, and the rule that an unreadable total requires review. The run record adds the extractor configuration, observed values, processing status and comparison outcome. A timeout is an execution failure, not a blank invoice total.
Give every input a stable identity
A filename alone is a weak identifier. Two files called invoice-final.pdf can contain different pages, while the same file can be renamed several times.
Assign a stable case ID and record a content checksum using a documented algorithm. Store the original test artifact separately from preprocessed versions. If you deskew an image, rasterize a PDF or split its pages, preserve the relationship between the source and the exact artifact sent to the system.
For a paired test, record the parent case and the changed condition. A readable document and its blurred counterpart should not look like unrelated records. That relationship lets you investigate whether a failure comes from the document's content, its layout or the imposed condition.
Keep identifiers free of sensitive personal information. Access to a manifest should not accidentally become access to a list of customer names or account numbers. Apply appropriate access and retention rules to inputs, labels and outputs, not just the source PDFs.
Version the answer and the comparison rule
A reference value needs context. Is 1,200.00 equivalent to 1200? Does a missing field mean null, an absent key or an explicit exception? Is a date normalized before comparison, and what happens when its interpretation is ambiguous?
Define these choices in a versioned extraction contract before comparing releases.
Keep raw observed text as well as any normalized form used for scoring. If a normalization rule changes, rerun both releases under the same rule or report the comparison as a changed evaluation method. Do not describe it as a model improvement.
Reference corrections should have a reason, reviewer and date. Correcting a mislabeled answer is good maintenance. Silently rewriting labels after seeing a disappointing score is not a defensible comparison.
Record enough configuration to repeat the run
For each run, capture the system version or available provider identifier, extraction schema version, instructions, preprocessing settings and relevant output options. Record the configuration you can actually observe; mark unavailable information as unknown.
Also capture the execution date, run ID, input manifest version, comparison-code version and any retry policy. If repeated runs can produce different outputs, preserve that variation rather than selecting the best answer. A single successful rerun should not erase an earlier failure.
This is a workflow design recommendation, not a claim that GritWorks automatically supplies a complete regression-management system. GritTransform supports configurable PDF/image extraction and saved API deployments. Your evaluation harness still needs to record which configuration was used and how its output was assessed.
Start with a compact case record
The following example is illustrative. It is neither a required standard nor a representation of a customer dataset.
- Case ID: invoice-017-blur
- Parent case: invoice-017-readable
- Purpose: invoice-total extraction under degraded input
- Artifact: controlled file location, checksum and page count
- Scenario: blurred copy; same fictional content as parent
- Reference version: invoice-reference-v3
- Contract version: invoice-schema-v2
- Expected result: the documented field values where source evidence is sufficient; review otherwise
- Comparison rule: exact currency match and declared amount normalization
- Owner: a named internal evaluation owner
- Data status: fictional test content; usage terms and sharing scope recorded
Attach run records separately: run ID, configuration, observed output, execution errors, per-field comparison and reviewer resolution. In a CSV, use linked tables rather than forcing every run into one widening row.
Compare the same population before making a release claim
Run the old and proposed configurations against the same frozen cases. Report added or removed cases separately. A higher score on an easier collection is not evidence of improvement on the original collection.
Inspect changes by scenario, not only an overall average. For example, a release may improve readable invoices while introducing more currency errors on photocopies. Decide in advance which failures block release, which require review and which can be accepted with an explicit owner and rationale.
Keep failed and excluded cases visible. List exclusions with reasons: unsupported document type, corrupted input, reference dispute or infrastructure failure. Do not quietly remove inconvenient examples from the denominator.
A small synthetic collection can reveal regressions, but it does not establish production error rates. Record the limits of your sample: document families, languages, layouts, input conditions and workflows not covered. NIST's AI RMF Core calls for documenting test sets, metrics and evaluation tools, and for assessing performance in conditions relevant to deployment. The manifest proposed here is one practical documentation pattern, not a certification or compliance guarantee.
Close the loop after a failure
A useful failure record ends with a disposition. Was the problem in extraction, the reference answer, the comparison rule, preprocessing or infrastructure? What changed, who reviewed it and which test proves the correction?
Add a new case when a genuinely new failure mode appears. Do not replace the old failing artifact with a cleaner version. Keep the difficult example, subject to its permitted use, so the same problem cannot disappear from future testing simply because the folder was tidied.
GritRender can generate controlled document variations with known values. Use those variations to expand a specific coverage gap, then check the generated artifacts and reference values before admitting them to the frozen regression set.
A practical first step
Choose one document workflow and a small approved set of cases. Make one person responsible for the reference data and another for reviewing unexpected differences where feasible. Freeze the inputs, document the expected behavior, run two configurations and investigate the changed cases.
If you cannot explain a difference without asking which PDF, which schema or which scoring script was used, the manifest is missing something useful. Add that information before adding another hundred documents.
Explore the free document dataset library for sample inputs, or discuss your document workflow with GritWorks. Start with a repeatable comparison, then expand the coverage that your actual release decision needs. Do not send sensitive production documents through public email or contact channels.
