Subset: turning LLMs into 26x faster decision-making machines

Published: September 25, 2026
By: Sparset Team

Many of the questions an application puts to a language model have a small set of possible answers. A model router needs one: which model should handle this request. A support workflow needs three: which department, whether a refund was asked for, how severe the issue is. A conventional integration still has the model write a full JSON response, one token at a time, to deliver a choice it could have made in a single step.

Subset is an experimental inference engine that skips the writing. The application declares its questions and the allowed answers. Subset runs an existing LLM, reads the model's scores for those answers, and assembles the structured output in code. It covers classification, routing, and rubric scoring, and it leaves the pretrained weights unchanged.

On our 250-case development benchmark with Qwen2.5-1.5B-Instruct, Subset reached a 167 ms median request latency, with 90% of outputs correct and schema-valid and 100% schema validity. By median latency that is 26.1x faster than unconstrained JSON generation from the same Qwen checkpoint and 2.4x faster than the Hugging Face implementation we compared against, on the same laptop GPU. These are results for this workload and this configuration, not general speedups. The benchmark methodology has the full setup.

Figure 1. Median request latency. Measured across 250 local development cases using the same Qwen2.5-1.5B-Instruct checkpoint on an RTX 3050 Laptop GPU with 4 GB VRAM. Lower is better. Model loading is excluded; each case was timed once per configuration, with configurations run in separate blocks.

The idea comes from two places. TypeSafe AI's Jev treats AI outputs as typed, probabilistic decisions and serves them from a purpose-built model, with a new architecture, parallel sampling, and reinforcement learning for calibrated decisions. The harshatheg/Qwen-2.5-1B-RLCD project on Hugging Face applies parallel constrained decoding to a conventional language model, and its approach was the direct inspiration for our implementation.

Subset is not affiliated with TypeSafe. It does not reproduce Jev's architecture or training, and we did not benchmark against Jev. Our question was narrower: how much faster can a pretrained LLM answer when the application already knows the possible answers?

From generating a response to scoring a decision

Take a support message:

I was charged twice. Please refund the duplicate charge.

The application defines a workflow with three questions: which department handles it, whether a refund was requested, and how severe the issue is. The department options might be billing, shipping, and returns, each with a one-line description of when it applies.

The engine receives that workflow and the message. It runs the model, reads the scores for the allowed answers, and normalises them within each field. Ordinary code maps the winning answers back to the workflow's labels and builds the response.

The model still does the interpreting. It reads the message and the instructions and decides between the options. What it no longer does is write. The application supplies the questions and answers; the engine does not discover a workflow on its own or split a conversation into tasks.

That boundary is what makes it usable inside software. A developer writes the rubric once in workflow.json, then sends fresh messages, context, or conversation history with each request. Changing the task means changing that file, not writing a GPU kernel.

On the default path the model sees each label and its description but answers through a single-token ID such as A, B, or C. Boolean questions use false and true. Compact answers let the engine read the relevant scores without generating an explanation or the JSON around it.

The resulting probabilities are uncalibrated model scores. A score of 0.8 does not mean an 80% chance of being right. Producing a valid response and making a correct, well-calibrated decision are separate problems, and this experiment addresses the first.

Where the work went

The weights are untouched. The engineering is in how a request is prepared, how independent questions share computation, and how scores become output.

Shared context. The common input prefix is processed once and its key-value cache is reused across the question branches.
Batched questions. Independent fields are evaluated together, with bounded batch sizes to control memory.
Selected answer scoring. For single-token answers, the model's final hidden state is projected onto only the output-head rows that matter, not the whole vocabulary.
JSON built in code. The response is assembled from allowed values. No token-by-token generation of keys, punctuation, or probability text.
Efficient attention and fused operations. PyTorch's scaled dot-product attention interface, plus optional Triton kernels for RMSNorm, SwiGLU, and rotary position operations.
Specialised branch handling. On supported configurations, one physical prefix cache is shared, attention is specialised for short branches, and questions of similar length are grouped to reduce padding.

Figure 2. Inference optimizations. Execution changes that reduce repeated processing and output-generation work while preserving the pretrained model weights. Specialized paths require compatible hardware, models, and workloads. The benchmark does not isolate each optimization’s contribution or measure scaling across multiple parallel questions.

Reusing computation and sharing memory are different things. Reusing a prefix cache avoids recomputing the input, but an implementation can still copy that cache for every branch. Our specialised attention path lets eligible branches read the same physical prefix cache while keeping their own question-specific data. Longer or unsupported workloads fall back to the ordinary path. Details are in the branch implementation notes.

The fused kernels combine neighbouring operations to cut intermediate memory traffic and launch overhead. Attention itself comes from PyTorch's SDPA interface alongside our branch kernel; this is not a rewrite of FlashAttention. These features have different compatibility requirements and are not universal go-faster switches. See the optimisation notes.

One limit to keep in mind: the headline benchmark asks one decision per case. It shows what the decision-scoring path does, not how the system scales across many parallel fields, and it does not separate how much of the gain comes from each kernel, prompt change, or avoided decoding step.

What we measured

Three execution approaches around Qwen/Qwen2.5-1.5B-Instruct, with Subset run both with and without CUDA graphs. FP16, on an NVIDIA RTX 3050 Laptop GPU with 4 GB of VRAM, under WSL on Linux.

The suite has 250 cases: 100 BANKING77 messages across ten intents, 50 constructed routing cases, 50 boolean judgments, and 50 templated severity cases. Some of these were used while developing the prompts, so this is a development-set evaluation, not a held-out test of generalisation.

All methods received the same inputs in the same order and used their own prompting and output procedures. The Hugging Face comparison used its pinned PyTorch and CUDA implementation, not its separately published Apple Silicon numbers.

Figure 3. Benchmark results. Results on the same 250-case development set. “Correct + schema-valid” requires both the expected decision and the complete requested schema. Base Qwen generated unconstrained JSON without repair; invalid outputs count as failures. These cases partly overlap prompt development and are not a held-out evaluation.

Local measurements from 18 September 2026. One timed request per case per configuration, model already loaded. Invalid and truncated outputs stay in the denominator and in the latency figures. The benchmark report has the full configuration and limitations.

Three things to read carefully in that figure.

What counts as correct. Correct and schema-valid requires both the expected choice and the complete requested output, including a valid probability distribution over all options. The baseline's 9.2% is therefore not Qwen's classification accuracy. It is how often unconstrained, greedy JSON generation with no repair step produced the right answer in the right shape. We did not measure the same gap against a grammar-constrained decoder or against other serving engines.

What the speedups are. The 26.1 and 2.4 figures are ratios of median request latency, not averages of per-case speedups. A request includes preparation, inference, result processing, and validation, and excludes model loading. Nothing here measures cold start, energy, or cost. Methods ran in separate blocks, so laptop power and thermal drift are possible confounders.

Where we lost. The engine beat the Hugging Face implementation by 15.2 percentage points overall, but not in every category.

Figure 4. Results by task. Correct and schema-valid outputs across banking intent (100 cases), model routing (50), boolean judgment (50), and severity scoring (50). Subset uses CUDA graphs disabled and trails the reference implementation on boolean judgments. Its 100% severity result applies only to these 50 templated cases.

The boolean regression is a reminder that optimising an overall number can leave individual tasks weaker. The perfect severity score is on 50 templated cases and says nothing about new inputs.

Two lessons

Prompts still matter when the answers are fixed

Restricting the output does not remove prompt sensitivity. The engine still has to make clear which text is the input, what question is being asked, and what each option means.

The current format separates input text from evaluation instructions and lists each option's label and description. We kept compact answer IDs after testing literal-label scoring: longer labels can need several tokens each, and the extra work did not improve results overall.

We also removed an experimental neutral-context correction. The engine no longer divides scores by a neutral-input distribution; it normalises over the allowed alternatives. And prompt boundaries make the task clearer to the model. They are not a defence against prompt injection. See the prompt format and the label-scoring notes.

CUDA graphs were slower on varied requests

CUDA graphs record a GPU operation sequence so it can be replayed with less CPU launch overhead. That pays off when input shapes repeat often enough to recover the recording cost.

On this varied-request benchmark they did not. The graph-enabled run captured 220 graphs and evicted 216, because too many requests needed fresh preparation. Turning graphs off cut median latency from 951 ms to 167 ms, and all 250 selected answers stayed the same.

The probabilities did not: six cases moved by more than one percentage point, with a maximum shift of 4.42 points. Identical choices do not imply identical confidence.

So CUDA graphs are optional and off by default. Workloads with repeated shapes may benefit, but that has to be measured on the real request distribution. A faster replay does not help if preparation makes the whole request slower.

The same request, three ways

The side-by-side video shows one routing request handled by Subset, the Hugging Face implementation, and ordinary Qwen generation. The input is "Implement a Java LRU cache", with coding as the expected route.

Subset and the Hugging Face implementation return the correct route in the required structure. Ordinary Qwen picks coding but wraps it in an extra object, which fails the schema. Decision correctness and interface correctness need separate checks.

The video replays recorded events at real speed. Each method ran separately, and the median of five warmed requests is shown. It is a chosen successful example, not an accuracy estimate. Its timings are separate from the 250-case benchmark, and token-streaming callbacks add overhead to the baseline recording. See the recording details.

Running it yourself

The code is in the repository. Define questions and allowed answers in workflow.json, then send changing context through the CLI or the Python API. Keep the engine loaded so model startup is not paid on every request.

The Qwen weights are not in the repository. On first use, Transformers downloads the model and tokenizer from Hugging Face and caches them locally. No fine-tuning is needed for the tested setup.

If you would rather have this running inside a production deployment than run it yourself, that is work we do. The evaluation is the same as for any other workload: we measure the current setup first and report the numbers with their conditions.

The validated checkpoint is Qwen2.5-1.5B-Instruct. The ordinary engine accepts compatible dense Qwen2, Qwen3, and Llama-style models, but each needs its own quality and performance checks. The custom kernels have a narrower tested envelope: Linux, CUDA, Qwen2. There is no support for arbitrary architectures, multi-GPU serving, or Ollama endpoints. See setup and compatibility.

What we measure next

Before drawing broader conclusions: a held-out dataset, repeated measurements under controlled power, stronger constrained-generation baselines, and workloads with several independent fields. Calibration needs its own evaluation. The current benchmark scripts depend on local frozen data and source snapshots, so a fresh clone is not yet a complete reproduction bundle.

The answer so far: a good part of Jev's idea does survive without a new model. Define the decisions an application needs, organise inference around those decisions, and an unchanged pretrained LLM gives faster responses and more usable structured output. What it does not give is calibration, and whether the model's judgments are good enough for a given workflow is a separate question. Valid JSON can still contain a wrong answer.

Credits and references

TypeSafe AI, Jev. The framing of AI as typed, probabilistic decisions comes from Introducing System One Models and Jev by Diogo Almeida. Subset makes no claim to reproduce RLCD or Jev's architecture.
Hugging Face, harshatheg/Qwen-2.5-1B-RLCD. The parallel constrained decoding project was the direct implementation inspiration and the comparison baseline. Revision 2af86848be75847ccb3553b0941cc51d6ef7e4e9.
Qwen. All three approaches used Qwen/Qwen2.5-1.5B-Instruct, revision 989aa7980e4cf806f80c7fef2b1adb7bc71aa306. The "1B" in the Hugging Face project's name does not change the tested model's 1.5B size.
Subset, the experimental engine. Code and setup, benchmark methodology, and optimisation details.