← All guides

RL environments for documents: building reinforcement learning environments with verifiable rewards

Reinforcement learning needs a signal it can trust. Documents generated with their field values recorded give you one: the model reads the page, returns structured output and code checks it against the answer key. This guide covers how to build that environment, design the reward and keep the model from gaming it.

THE SHORT ANSWER

An RL environment is the task, observations, actions and reward function a model learns from by trial and error. Documents make good reinforcement learning environments because synthetic forms carry known field values, so code can score each output against the recorded ground truth after normalization, with no human grader or judge model in the loop.

How this page was written. GritWorks makes GritRedact, GritTransform and GritRender, so we are not neutral. What we say about other products reflects their own public websites and documentation as of 21 November 2026. We make no claims about other products' accuracy. Products change: confirm details with each vendor.

Reward functions for document tasks at a glance

Most document rewards combine a few programmatic checks, each computed per field against the recorded ground truth. Each one rewards something different, and each can be gamed on its own.

RewardWhat it scoresUse it forWatch out for
Exact match after normalization1 when the normalized prediction equals the normalized true value, otherwise 0Identifiers, dates, amounts and codes, where a near-miss is still wrongA normalizer so loose that wrong values match, or so strict that correct ones fail
Normalized character error rate (partial credit)1 − min(1, edit distance ÷ length of the true value)Names, addresses and free text, and early training when exact matches are rareLong outputs that contain the answer somewhere; cap value length or pair with exact match
Field coverageShare of the fields on the page that the output fillsTeaching the model to find every fieldGuessing to fill every key, unless paired with a hallucination penalty
Hallucinated-field penaltyA deduction for values filled where the ground truth says the field is blank or absentStopping the model from inventing valuesGround truth must record blanks explicitly, or correct abstentions are punished
Schema validityOutput parses and has the expected keys and typesA gate before content is scoredValid but empty output, if it is the only check
Grounding (box overlap)Overlap between the region the model cites and the field's bounding boxTasks that must show where a value came fromNeeds boxes in the ground truth; reward it only when the value is also right

What an RL environment is

In reinforcement learning a model, the policy, takes an action, receives a reward and updates so that it earns more reward next time. The RL environment is everything outside the policy: it presents an observation, accepts the action, decides when the episode ends and returns the reward. For language and vision-language models, reinforcement learning environments are usually packaged tasks: inputs, the actions the model may take and a grader.

  • Observation. What the model sees: a page image, a PDF's text layer, OCR output, or a combination.
  • Action. What the model produces: structured output such as JSON fields, or, in a multi-step environment, tool calls such as zoom, crop, rotate or run OCR before it answers.
  • Reward. A number computed from the action, ideally by code, that says how good the answer was.
  • Episode and reset. One document, or one packet of related documents, per episode, with a new document drawn on reset.

An environment differs from a dataset because it is interactive and scored as the model acts, and from an eval only in purpose: the same task and grader can serve both, as long as documents used for evaluation never reach training.

Why documents suit verifiable rewards

A verifiable reward is one a program can compute from the output, without a human grader or a judge model. Mathematics and code have them: the answer checks, or the tests pass. Document understanding has them too, whenever the value of every annotated field is known in advance.

  • The answer key exists before the model runs. A synthetic invoice, bank statement or claim form is generated from values chosen first, so totals, dates and account numbers are recorded rather than labelled afterwards by someone reading the page.
  • Scoring is per field. A form has many fields, so the reward is graded rather than pass or fail, and every failure points at a specific field.
  • Difficulty is a dial. Layout, field count, handwriting, script and capture condition can change while the values stay fixed, which makes a curriculum possible.
  • Boxes make grounding checkable. When the ground truth records where each value sits, the reward can require the model to point at the right region, not just produce the right string.
  • No real people. Fictional records keep personal data out of the training loop and out of the rollouts that get logged and inspected.

Real documents can feed an environment too, but someone has to label them first, and human labels contain errors that a reward then teaches as truth. Where real documents are used, remove personal data before they enter the pipeline (see what redaction is). The synthetic test data guide covers what makes generated documents realistic enough to learn from.

How to design a reward for document extraction

  1. Normalize both sides the same way. Trim and collapse whitespace, apply one Unicode form and one case rule, convert dates to ISO 8601 and amounts to a plain decimal without currency symbols or thousands separators. Keep identifiers exact apart from separators you agree to ignore.
  2. Choose the check per field type. Exact match after normalization for identifiers, dates, amounts and codes; normalized character error rate for names, addresses and free text, where a near-miss deserves partial credit.
  3. Score what is missing and what is invented. Reward coverage of the fields the page actually has, and deduct for values the output fills where the ground truth says the field is blank or absent.
  4. Aggregate to one number between 0 and 1. Average the per-field scores, optionally weighted by how much each field matters downstream, and gate the total on schema validity.
  5. Log the components, not only the total. Exact match, partial credit, coverage and hallucinations per field, condition and script. The total trains the policy; the components tell you what it learned.

Dense, partial-credit rewards help early, when exact matches are rare. Shift weight toward exact match as the policy improves, because downstream systems need the value right, not close.

Reward hacking, and how to guard against it

Reward hacking is a policy raising its reward without getting better at the task. Programmatic document rewards are easier to audit than judge-model scores, but they can still be gamed:

  • Filling every field. If only coverage is rewarded, the model guesses. Pair coverage with the hallucination penalty and record blank fields explicitly.
  • Copying the page into each field. A substring or “contains” rule rewards dumping all the text. Score exact match after normalization, and cap or penalize value length.
  • Exploiting the normalizer. If normalization strips too much (all punctuation, all spaces, the leading zeros of an account number), different values become equal. Test the normalizer with near-miss pairs that must not match.
  • Memorizing templates. With few layouts, the model learns where a value usually sits instead of reading it. Randomize values, positions and layouts, and hold out whole templates.
  • Memorizing documents. If the same documents recur across episodes, the model can learn their values. Generate fresh documents, or draw from a pool much larger than the number of training steps.
  • Reading a side channel. PDF text layers, metadata, file names and generator marks can leak the answer. Render observations to images, strip metadata and use opaque file names.
  • Gaming a judge model. If part of the reward comes from another model, the policy can learn that model's preferences. Keep programmatic checks at the core and use judges only for what code cannot check.

Watch for divergence: if training reward climbs while exact match on a fixed held-out set stalls, inspect the rollouts. Read a sample of outputs at every checkpoint, not only when the curve looks wrong.

Goods invoice: readable sample from the OCR pack

FREE SAMPLE PACK · OCR

Try it with the free OCR sample pack.

Fictional documents, clean and degraded, with the expected value and position of selected populated fields.

10 PDFs · Purchase order, Goods invoice, Services invoice and more · JSON and bounding boxes · CC BY 4.0

Free sample · No call required · No platform account

Need larger volumes, your own layouts or other formats? Talk to us ↗

A curriculum by degradation: clean, blur, photocopy, fax

The same document can be rendered clean and then degraded: blurred, skewed, photocopied, faxed, or photographed at an angle or in low light. Because the field values do not change, the reward stays valid at every stage and only the difficulty moves.

  1. Start clean. Digital PDFs or high-quality scans, so the model learns the task and the output format.
  2. Add blur and skew. Mild degradation that keeps most characters legible.
  3. Move to photocopies. Harsh contrast, uneven toner and specks.
  4. Finish with fax. Low resolution, binarisation and scan-line streaks.

Promote the policy to the next stage when its reward plateaus, keep a share of earlier stages in every batch so it does not forget them, and report reward per condition. On the public degraded-scan OCR benchmark, built on fictional sample documents, fax was the hardest synthetic condition for every engine tested, which is why it belongs at the end of the curriculum rather than the start.

Multilingual and Indic-script documents

Documents in Hindi, Marathi, Bengali and other Indic scripts are where verifiable rewards need the most care, and where models have the most to learn. In the same benchmark's first pass, no CPU engine reached 60% field exact match on the clean Hindi, Marathi or Bengali forms, and on fax every engine scored 12.2% or lower.

  • Normalize Unicode first. The same Devanagari or Bengali word can be encoded as different code-point sequences. Apply one normalization form (NFC is common) to prediction and ground truth before comparing.
  • Decide on joiners and digits. Zero-width joiners and non-joiners change rendering but rarely meaning, and native digits and ASCII digits can both be correct. Write the rule down and apply it to both sides.
  • Measure character error on grapheme clusters. Counting code points over-penalizes errors in conjuncts and vowel signs; counting the characters a reader perceives gives fairer partial credit.
  • Fix the target script. State whether the expected value is in the original script or transliterated, and never mix the two in one field.
  • Report per script. A blended score hides a script the model cannot read.

Keeping eval sets out of training

  • Split by generator, not by page. Pages from the same template, seed or value pool share structure. Hold out whole templates, layouts and value pools for evaluation, not random pages.
  • Hold out conditions and scripts too. Keep at least one layout family and one degradation unseen in training, so the eval measures transfer rather than recall.
  • Fingerprint everything. Hash each document and its field values, check training batches against the eval set's hashes, and catch near-duplicates with perceptual image hashes.
  • Keep eval sets private and versioned. Anything published can end up in a crawl. Version the set and record which checkpoints have seen what.
  • Keep a real held-out set. Synthetic documents train and test the skill; a small set of real documents, with personal data removed, tells you whether it transfers to the documents you will actually see.

Sample documents, the benchmark and GritWorks

  • The free document dataset packs are a way to prototype a grader: each sample has 10 PDFs of fictional documents, five clean and five degraded, with structured JSON and bounding-box JSON for selected populated fields, licensed CC BY 4.0 (third-party marks excluded). They are a testing resource, not a free tier, and not a complete training corpus.
  • The degraded-scan benchmark publishes its code, data and per-condition results.
  • GritRender generates synthetic documents with known values for training, testing and evaluation, including handwriting, stains, fax artifacts, photocopies and phone-photo conditions. It runs offline in your environment or as sales-assisted SaaS, and pricing is agreed with sales. See document data for frontier AI labs.
  • For real documents, GritRedact removes personal data before they are used. The AI training data guide compares collecting, licensing, labeling and synthesizing data.

What GritWorks does not replace

  • GritWorks complements authorization, governance and e-discovery tools rather than replacing them. Access control, data catalogues, retention policy and legal review platforms keep their jobs; GritWorks prepares the documents, images and audio that move between them.
  • Redaction is not authorization. If a person or system is not entitled to a record, block access instead of sending a redacted copy.
  • Automated redaction alone is not a declaration of HIPAA de-identification. HIPAA recognizes the Safe Harbor and Expert Determination methods; policies, validation and human review must match the intended use and recipient.
  • PII flagging is not redaction. GritTransform can flag potentially sensitive values in extracted results, but that does not remove them from the source. Use GritRedact when content must be removed.
  • Engagement is sales-assisted. GritWorks has no public or self-serve API, SDK, MCP server or free tier, and publishes no price list: pricing is agreed with sales. Customer APIs are provided after onboarding.

Common questions

What is an RL environment?

An RL environment is everything a model interacts with during reinforcement learning: it presents an observation, accepts the model's action, ends the episode and returns a reward. For language models it is usually a packaged task with inputs, allowed actions and a grader.

What are reinforcement learning environments for documents?

Tasks in which a model reads a document, such as an invoice, bank statement or claim form, and returns structured output that code scores against the document's known field values. The same documents and grader can also serve as an eval, if they are kept out of training.

What is a verifiable reward?

A reward a program can compute from the model's output without a human grader or a judge model, such as a math answer that checks or code whose tests pass. Documents generated with recorded field values allow verifiable rewards for extraction.

How do you design a reward for document extraction?

Normalize prediction and ground truth the same way, use exact match for identifiers, dates and amounts and normalized character error rate for free text, reward field coverage, penalize invented values, and aggregate to a score between 0 and 1.

What is reward hacking, and how do you prevent it?

Reward hacking is a policy raising its reward without improving at the task, for example by filling every field or copying all the page text. Pair coverage with penalties, avoid substring matching, vary templates, block side channels and track a fixed held-out set.

How do you keep eval documents out of training?

Split by template, value pool and condition rather than by page, fingerprint documents and check training batches against the eval set, keep eval sets private and versioned, and keep a small real held-out set with personal data removed.

Are the free dataset packs an RL environment?

No. Each pack has 10 fictional PDFs with reference JSON and bounding boxes for selected fields: enough to prototype a grader, not a training corpus. The packs are a testing resource, not a free tier.

Goods invoice: readable sample from the OCR pack

FREE SAMPLE PACK · OCR

Try it with the free OCR sample pack.

Fictional documents, clean and degraded, with the expected value and position of selected populated fields.

10 PDFs · Purchase order, Goods invoice, Services invoice and more · JSON and bounding boxes · CC BY 4.0

Free sample · No call required · No platform account

Need larger volumes, your own layouts or other formats? Talk to us ↗

A USEFUL START, IN YOUR INBOX

Your next dataset.
Delivered.

10 PDFs. Clean and challenging versions. JSON and bounding boxes included.

Use the inbox where you’d like to receive your pack.

Free sample · No call required · No platform account

Start with your workflow

Put your unstructured data to work.

Tell us what your team needs to achieve. We’ll help you choose the right products, work through your requirements and explore on-premises or SaaS deployment.

Request a working session