A document assistant can find the right files and still give the wrong answer. It might combine figures from incompatible records, miss an exception in a policy, or reach the right conclusion without citing the evidence that supports it. Retrieval gets information into the context window. Making responsible use of that information is a separate problem.
We built Synth, a document-synthesis fine-tune of Qwen3.5-4B, to work on that second problem. On our internal, synthetic 500-case evaluation, its composite score increased from 91.16 to 95.22 out of 100. The biggest gains were in cross-document calculation and insufficient-evidence handling. There were regressions too, and a separate visual extraction benchmark showed no improvement.
The finding is specific: this checkpoint improved several aspects of answering from supplied text. It does not establish better retrieval, better OCR, or a better end-to-end document agent.
The work after retrieval
Consider a question about whether a customer qualifies for a service. The answer may depend on an agreement, a later amendment, and an account record. Finding all three documents does not settle which rule applies. The model must reconcile them, identify missing facts, and explain its decision without silently filling the gaps.
Synth's training objective covers those behaviors: combining evidence, resolving source conflicts, performing grounded calculations, following conversational references, and recognizing when the supplied information is insufficient. We wanted a model that could work with different document presentations rather than depend on one particular retrieval wrapper.
The released dataset contains 9,878 records, split into 9,278 training examples, 287 validation examples, and 313 test examples. It combines NVIDIA's ChatQA training data with our own synthetic enterprise scenarios. Its construction includes direct supervised examples and teacher-rewritten answers, with evidence presented as plaintext chunks, Markdown sources, JSON tool results, and XML documents.
These formats change how information is presented, not what the model is allowed to invent. A response should remain grounded in the evidence whether that evidence arrives as a numbered passage or a tool result. The dataset validation found no exact benchmark overlap; that check is narrower than proving the absence of all semantic contamination.
A small adaptation with a defined target
Synth uses LoRA: small trainable adapters applied to the base model rather than updating every weight. We trained for one epoch with rank-32 adapters, BF16 base weights, and a learning rate of 0.00004. The loss is applied to the final assistant completion, while the prompt and supplied evidence are masked. That directs the learning signal toward the answer we want the model to produce.
The adaptation targets the language model and preserves the vision tower. Keeping that component intact does not guarantee that visual document behavior stays unchanged: extraction still depends on the language model interpreting the image-derived information and producing the requested structure. The released model card records the training recipe and available artifacts.
This recipe is one tested approach, not evidence that a single epoch or a particular adapter rank is optimal. We have not isolated those choices in an ablation study.
What changed in the answers
To separate synthesis from retrieval, we supplied both models with the available documents. They answered the same 500 cases across ten task types and ten enterprise-domain settings, with the same prompt profile and generation limits. Thinking was disabled for both models.
The distinction between a correct conclusion and a useful, well-supported response matters here:
The content-check pass rate improved by just 0.80 percentage points, four additional passes out of 500. The larger composite gain reflects improvements in other required behaviors as well. Citation F1 measures how well the cited document IDs match the reference evidence set; it does not independently verify every claim in the answer. Response status distinguishes a supported answer from insufficient evidence or a need for clarification.
Cross-document calculation rose from 85.30 to 98.70, and insufficient-evidence handling from 75.50 to 94.60. For the latter, both models already passed the content check on all 50 cases. The difference was in satisfying the remaining response requirements, not a newly demonstrated ability to recognize 50 previously unrecognized evidence gaps.
Cross-document eligibility remained the weakest slice: its composite score increased from 67.53 to 71.17, but only 30 of 50 Synth answers passed the eligibility content check, compared with 28 for the base model. Latest-version precedence fell from 91.67 to 88.33. Those results matter for workflows where a missed exception or an outdated rule can change a decision.
Synthesis is not visual extraction
We also compared the checkpoint with the base model on ExtractBench, which measures schema-conditioned structured extraction from document images. Across 370 paired documents, base-model F1 was 80.00, compared with 79.14 for Synth. The paired difference was −0.86 percentage points, with a 95% bootstrap interval of [−2.93, +1.19]. This test did not establish an extraction improvement.
Both models completed all 370 cases after symmetric infrastructure repair: the union of 13 documents affected by transient serving failures was retried for both models without changing the task or generation budget. Seventeen of the twenty long documents had pages omitted by the shared context limit, so those results do not represent full-document extraction.
This is a useful boundary for the release. Better answers from supplied text do not automatically transfer to extracting structured records from images. The synthesis test also leaves real retrieval, longer inputs, and end-to-end agent behavior unverified.
Try Synth within its tested scope
The weights are available as Sparset/Synth-v1-qwen3.5-4b. The repository was first published as Mintzs/ora-qwen3.5-4b-docsynth-v1; Synth is the current model name. The release includes merged BF16 SafeTensors, the LoRA adapter, and a bitsandbytes INT8 variant. The results above should not be assumed to hold unchanged for the quantized variant.
This is a research-only, noncommercial release, reflecting restrictions in the training-data mixture. It is not a drop-in commercial deployment recommendation. Start with representative examples from your own workflow, compare against the base model, and test the failure modes that matter most, especially eligibility decisions and version precedence.
Build a model around your workflow
Synth illustrates the work behind a useful task model: define the behavior, train for it, and measure both improvements and regressions. With Sparset Forge, we help teams develop personalized models around their own data, evaluation requirements, and infrastructure. The results from this research checkpoint do not guarantee an outcome for a different workflow.
If you want a custom document model, or a model running on your own infrastructure, book a meeting.
Technical appendix
The synthesis evaluation used ora_enterprise_rag_agent_500 version 1.0.0, an internal synthetic set with 50 cases per task and 50 per domain. Both models produced outputs for all 500 matching case IDs; none were excluded from the comparison. The oracle-context profile allowed 4,096 input tokens and 320 new tokens, with deterministic generation and enable_thinking=false. Each model ran on a Tesla T4 with batch size four. This text-only evaluation does not support a controlled latency claim.
The composite is: 45% content-check score + 25% citation F1 + 15% response-status correctness + 10% JSON compliance + 5% citation validity. Citation validity checks whether cited IDs exist among the supplied documents; both models scored 100% on that check. Citation F1 additionally rewards selecting the reference evidence set. Content checks use task-specific deterministic rules rather than independent human assessment. No confidence interval was reported for the composite.
These are corrected scores. As first run, the automated scorer gave composites of 77.80 for Qwen3.5-4B and 82.15 for Synth. Reviewing the raw outputs, we found rules that penalized acceptable answers: an answer that mentioned a listed phrase, such as a superseded value or the owner named in a non-authoritative record, scored zero even when its conclusion was correct; answers and citations inside a slightly different JSON wrapper were treated as empty; and eligibility answers were matched by keyword rather than judged by their stated eligible or not-eligible decision. We corrected these rules and re-scored both models’ saved outputs identically, without rerunning either model. The nonstandard wrapper is still penalized under JSON compliance. The corrections raised both models’ scores and narrowed the overall gap from 4.35 to 4.06 points. Because they were made after inspecting the outputs, this evaluation is diagnostic rather than preregistered, and the original scorer’s confidence interval does not apply to the corrected scores.
For the underlying model and release information, see the Qwen3.5-4B model card and Synth model card.