A mortgage note, an application and an appraisal can each look convincing while describing three different loans. The borrower name drifts between files. A property identifier changes. An amount is copied from a different scenario. A document-processing system may extract every visible value correctly and still assemble an unusable case.
For teams testing multi-document workflows, the useful unit is the complete file. Start with a small set of fictional entities and relationships, map them into each document, and specify which differences should trigger review. Then vary the appearance of those documents without accidentally changing their meaning.
This guide proposes a practical test-design method. Its examples are fictional and are not underwriting rules, legal advice or claims about a particular customer's system.
Start with one fictional case record
Before generating PDFs, write a compact case record that defines the facts shared by the file. Give each entity a stable test identifier rather than using a person's display name as the link.
For an illustrative case, define:
- Case: TEST-LOAN-042, containing one applicant, one property and one proposed loan.
- Applicant: PERSON-017, displayed as Morgan Ellis. Keep a separate mapping for any permitted abbreviated name.
- Property: PROPERTY-009, represented by a clearly fictional test address and an internal property identifier.
- Loan: LOAN-042, with an illustrative principal of USD 240,000 and a defined document-effective date.
- Documents: application version A, note version A and appraisal version A, each associated with the same case and its own document identifier.
The exact identifiers and amounts are unimportant. Their relationships are the point. Generate the applicant once for the scenario, then reuse that entity wherever the applicant appears. Do the same for the property and loan. Generating each document independently and hoping names happen to match makes failures hard to interpret.
The database concept is familiar: a foreign key connects a record to a valid referenced record. PostgreSQL's constraint documentation distinguishes these relationships from uniqueness and other field constraints. A PDF does not enforce such relationships by itself. Your generation and validation workflow needs to preserve and check them.
Map meaning, not just matching strings
For each document, list the source entity, target field and permitted presentation. An application might display a full applicant name while another document uses a middle initial. Currency may appear with a symbol, a currency code or a thousands separator. Those differences can be legitimate.
Define the comparison rules before creating the test set:
- Exact identity: internal case, entity and document identifiers must link to the intended records.
- Normalized equivalence: formatting differences such as USD 240,000 versus 240000 may represent the same amount when currency and units are specified.
- Role-specific meaning: mailing address and subject-property address need not be equal. Do not create a rule that falsely treats every address difference as an error.
- Time-specific meaning: a later document can legitimately supersede an earlier value. Record effective dates and which version the workflow should use.
- Allowed absence: a field absent from one document should not be silently filled from another unless the workflow explicitly permits that operation and retains its source.
Keep this mapping reviewable by both an engineer and a domain owner. A technically consistent dataset can still encode the wrong business meaning. If the team cannot explain why two values should match, it is too early to make equality a release gate.
Build a valid baseline before breaking it
Create one small, internally consistent file and inspect it end to end. Check that the rendered pages contain the values intended by the case record. Confirm that repeated values fit their fields and are not clipped, wrapped into the wrong row or obscured by an unrelated element.
Keep three forms of evidence together: the case record, the rendered documents and the expected downstream result. The expected result should describe the workflow behavior, not merely repeat every source field. For a baseline reconciliation test, that might mean “associate all three documents with TEST-LOAN-042 and report no configured identity or principal mismatch.” It should not mean “approve this loan.”
If extraction is part of the workflow, compare extracted values with the known source values before diagnosing a reconciliation defect. A wrong character in an identifier and a correctly extracted but contradictory identifier are different failures, with different owners. The field-level OCR evaluation guide covers that earlier measurement layer.
Introduce one deliberate contradiction at a time
Once the baseline works, clone the case and change a controlled relationship. Keep the intended change in a separate scenario record so that a generated defect is distinguishable from an accidental one.
Useful fictional scenarios include:
- Wrong applicant: one document references PERSON-018 rather than PERSON-017. Expected behavior: flag the identity conflict rather than silently merge the records.
- Wrong property: the appraisal belongs to a different property while the loan identifier remains unchanged. Expected behavior: surface the inconsistency for review.
- Amount mismatch: change the note's principal to USD 245,000 while the application remains USD 240,000. Expected behavior: identify the two values and their sources under the configured comparison rule.
- Stale version: include an older application plus its replacement. Expected behavior: apply the documented version policy or request review when precedence is unclear.
- Missing document: omit a required document from the test file. Expected behavior: report incompleteness rather than invent the missing evidence.
- Duplicate submission: submit the same document twice with different filenames. Expected behavior: follow the agreed duplicate policy without counting it as two independent sources.
These are candidate test cases, not universal banking requirements. A business owner must specify the correct response for the actual application. Initially change one variable per scenario. Later, combine selected failures to test precedence and error reporting once individual causes are understood.
Vary appearance without losing the reference answer
A consistent file can be rendered in clean, scanned or degraded forms to test whether the downstream workflow remains stable. Keep the logical case record unchanged when the test is about appearance. When the test is about a contradictory fact, record that fact change explicitly.
This separation helps isolate whether a failure comes from visual recognition, field interpretation, document association or business-rule handling. It also prevents “more realistic” documents from quietly changing the expected answer.
Some transformations can make a field genuinely unreadable. In that case, preserving the original value in a reference record does not mean the system should guess it. Mark whether the expected response is a value, an uncertainty signal or a review request. Have a reviewer confirm that the rendered evidence supports the test's intended difficulty.
Synthetic variation also has limits. Templates can miss real layout diversity and unusual combinations. Include permitted, reviewed real-world examples where available, and keep a separate evaluation set from material used to tune the extraction or reconciliation workflow.
Keep a manifest and a release checklist
A simple manifest can connect every artifact to its purpose. Record the case and document identifiers, scenario name, template version, generation configuration, expected values, intentional contradictions and expected workflow response. Preserve the actual generated files as well as their identifiers; a random seed alone may not reproduce output after a generator or template changes.
Before accepting a test pack, check:
- Every document links to the intended case and entity records.
- Baseline relationships pass, while intentional conflicts remain visible.
- Dates, amounts, units and version rules are defined rather than inferred.
- Rendered pages match the reference record, including repeated fields.
- Missing or unreadable evidence produces the agreed exception behavior.
- A reviewer can trace a reported failure back to the exact input and rule.
Start with enough cases to exercise the critical relationships, then expand based on observed gaps. Hundreds of near-identical files add less diagnostic value than a small set whose expected outcomes are unambiguous.
Where GritWorks fits
GritRender supports configurable values, rules and document variations, including consistent content across related files. GritWorks' public loan-document testing case study describes a redact-then-generate workflow that retains useful layouts and uses synthetic borrower, property and loan values.
When starting from production documents, establish permission for the intended use and review the source material before using a sanitized derivative as a template. Redaction is one preparation step, not a blanket authorization to redistribute the source or its derivative.
The manifest, reconciliation rules and release checklist above are a recommended evaluation design, not a claim that every check is automatically enforced by a product. Bring your document types, cross-document relationships and expected exceptions to a GritWorks working session. The first useful deliverable is a small file whose facts and expected behavior your team can explain and verify.
