← All comparisons

Synthetic document data vendors: a checklist for realism, ground truth, leakage and edge cases

Synthetic documents are only useful if your system behaves on them the way it behaves on real ones. This checklist covers what to ask a vendor and how to verify the answer yourself.

THE SHORT ANSWER

Evaluate a synthetic document data vendor on six things: realism of layouts and values, ground truth for every field, leakage and memorization risk from any real seed data, coverage of edge cases, controllable degradation such as blur, fax noise and handwriting, and output formats. Ask for a labelled sample, run it through your own pipeline, and compare results.

How this page was written. GritWorks makes GritRedact, GritTransform and GritRender, so we are not neutral. What we say about other products reflects their own public websites and documentation as of 17 November 2026. We make no claims about other products' accuracy. Products change: confirm details with each vendor.

Four ways to get synthetic documents, compared

Approaches across, properties down. The last column is GritRender, the GritWorks generator.

FeatureDe-identify and replaceText or tabular generator with exportBuild it yourselfRule-driven generator: GritRender
Adds edge cases production never gave youNoPartialContent, not page conditionsPartialIf you build themYes
Controlled degradation with clean twinsNoNot typical of this approachNoNot typical of this approachPartialAugmentation librariesYes30+ negative document variants
Field-level ground truth recorded at generationPartialReplaced values are known; the rest is unlabelledPartialDepends on the toolPartialIf you build itYes
Low engineering effortYesYesNoYes
Runs inside your networkPartialSome tools self-hostPartialDepends on the toolYesYesOffline on your infrastructure

✓ Yes✗ No◐ Partial

The evaluation checklist

Use the same questions for every vendor, including an in-house generator. Score each on the evidence it gives you, not on the demo.

CriterionWhat to askHow to verifyRed flag
RealismDo layouts, fonts, field formats and value distributions match the documents your system will see?Run vendor samples and a few approved real documents through your pipeline; compare error patterns by field.Only clean, digitally born PDFs; values that ignore real formats (dates, IDs, currency).
Ground truthDoes every document ship with the correct value for every field, plus coordinates?Check that the JSON reconciles with the page: totals add up, IDs match across related documents, boxes land on the text.Labels produced by running OCR on the output, rather than recorded at generation.
Leakage and memorizationWas any real data used as a seed or to train the generator? How do you stop real records reappearing?Ask for the seed policy and for near-duplicate checks against the seed set; search outputs for known real identifiers.A model trained on customer documents with no description of how outputs are checked.
Edge casesCan you request rare layouts, missing or conflicting fields, and business-rule failures on purpose?Ask for a set built from your rules, including negative cases, with the expected outcome for each.Variation only in values, never in structure or business logic.
Controlled degradationCan you set blur, skew, low light, fax and photocopy noise, stains and handwriting per document, and keep a clean twin?Request readable and degraded pairs of the same record so you can isolate the effect of image quality.Random noise you cannot reproduce or turn off.
Formats and scalePDF, TIFF, PNG and JPEG? Multi-page files? Audio or images if you need them? Volumes you need?Generate a pilot batch in your target formats and load it into your systems unchanged.A format conversion step you have to build.
Deployment and governanceWhere does generation run, and what is retained? Can it run inside your network?Review the data flow, retention and any outbound connections.Seeds or prompts sent to a third-party service you did not approve.

Four kinds of synthetic document source

  1. De-identification with synthetic replacement. Tools that take your real files and swap sensitive values for realistic fake ones, such as Tonic Textual. You keep real structure and variety; you also keep real data in the pipeline until it is transformed, and you get no new edge cases.
  2. Text and tabular generators with document export. Tools that generate records or free text and export them to PDF or Word, such as Tonic Fabricate. Good for content; check whether layouts, scans and image conditions match your inputs.
  3. Template- and rule-driven document generators. Tools that render documents from layouts, field rules and degradation settings, recording the true value of every field as they go. GritRender is in this group.
  4. Build it yourself. Templates, a PDF library, an image-augmentation library and, increasingly, an LLM to write values. Cheap to start; the cost is keeping labels exact, values consistent across related documents, and the generator maintained.

Leakage and memorization: the question most evaluations skip

"Synthetic" does not by itself mean "contains no real data". Risk enters in two places:

  • Seeds. If real documents are used as templates, any value not replaced travels into every output. Ask how seeds are reviewed and sanitized first. A common pattern is to redact a reviewed set (for example with GritRedact) and use only that output as a template.
  • Trained generators. If a generative model was trained or fine-tuned on real documents, it can reproduce fragments of them. Ask what it was trained on, and how outputs are checked against the training set for near-duplicates.

Your own check is simple: plant a few unique canary values in any seed you provide, and search the generated set for them.

Run a two-week pilot

  1. Pick one workflow (for example, claims intake or loan documents) and list its document types, required fields and business rules.
  2. Ask each vendor for the same small set: readable and degraded versions of each document type, with field-level JSON and coordinates.
  3. Run the set through your OCR or extraction pipeline and a small approved real sample; compare field accuracy by condition.
  4. Ask for one change (a new field, a new degradation, a rule violation) and time how long it takes.
  5. Review the deployment, seed and retention answers with your security team.

The free GritWorks sample packs are built this way: ten fictional PDFs per pack, five readable and five degraded, with JSON and bounding boxes for selected fields. Use them to rehearse the pilot before you talk to any vendor. Our insurance claims test data guide walks through the same method for one workflow.

When to choose which

  • De-identification with replacement when you have enough real documents and mainly need them safe to use.
  • Text or tabular generators when the content matters more than the page image.
  • A rule-driven document generator such as GritRender when you test OCR, extraction or document AI and need layouts, degradation and a known answer for each field, including cases production never gave you.
  • Build it yourself when you have one or two simple document types and an engineer to maintain the generator.

What GritWorks does not replace

  • GritWorks complements authorization, governance and e-discovery tools rather than replacing them. Access control, data catalogues, retention policy and legal review platforms keep their jobs; GritWorks prepares the documents, images and audio that move between them.
  • Redaction is not authorization. If a person or system is not entitled to a record, block access instead of sending a redacted copy.
  • Automated redaction alone is not a declaration of HIPAA de-identification. HIPAA recognizes the Safe Harbor and Expert Determination methods; policies, validation and human review must match the intended use and recipient.
  • PII flagging is not redaction. GritTransform can flag potentially sensitive values in extracted results, but that does not remove them from the source. Use GritRedact when content must be removed.
  • Engagement is sales-assisted. GritWorks has no public or self-serve API, SDK, MCP server or free tier, and publishes no price list: pricing is agreed with sales. Customer APIs are provided after onboarding.

Common questions

What should I look for in a synthetic document data vendor?

Realistic layouts and values, exact ground truth for every field, a clear answer on leakage and memorization risk, edge cases you can request, controllable degradation with clean twins, the formats your systems take, and a deployment model your security team accepts.

What is ground truth in synthetic documents?

The correct value of every field, recorded when the document is generated rather than read back from it, usually as JSON with the field's position on the page. It lets you score OCR and extraction output automatically.

Can synthetic documents leak real data?

Yes, if real documents are used as seeds without sanitizing them, or if a generator trained on real documents reproduces fragments. Ask vendors how seeds are reviewed and how outputs are checked against training data, and plant canary values to test it yourself.

Why do degraded documents matter?

Production documents arrive as phone photos, faxes, photocopies and skewed scans with handwriting and stamps. A test set of clean PDFs overstates accuracy. Readable and degraded pairs of the same record show how much image quality alone costs.

Is synthetic data enough to validate a document AI system?

Not on its own. Synthetic data predicts real performance only when its structure, media characteristics and distributions are validated against the target workflow. Keep a small approved real sample to check that results on synthetic data hold.

How does GritWorks provide synthetic document data?

GritRender generates documents, images and audio from templates, field rules and degradation settings, with known values for each field. It runs offline on your infrastructure or as sales-assisted SaaS. Free sample packs are available by email from the dataset library; larger datasets are scoped on a call.

Start with your workflow

Put your unstructured data to work.

Tell us what your team needs to achieve. We’ll help you choose the right products, work through your requirements and explore on-premises or SaaS deployment.

Request a working session