Every time someone asks a chatbot a question, generates an image, transcribes a call, or gets a suggested reply in their inbox, a computer somewhere runs an AI model to produce the answer. That step is called inference.
It is the part of AI most people use and the fewest people talk about. It is also where most of the money goes once a product has real users. This guide covers what inference is, how it works, what it costs, and how teams make it faster and cheaper.
What is inference?
Inference is using a trained AI model to produce an output from a new input. You send the model something (a question, an image, a recording), the model runs, and you get something back (an answer, a label, a picture, a transcript).
The word comes from statistics, where it means drawing a conclusion from data. In AI it simply means running the model, as opposed to building it.
What is the difference between training and inference?
Training builds the model. Inference uses it.
Training means showing a model a huge amount of data and adjusting it, step by step, until it gets good at a task. It takes a lot of computing power, often thousands of GPUs for weeks, but it happens once, or once in a while when the model is updated. Fine-tuning is a smaller version of the same thing: taking a trained model and adjusting it further on a narrower set of data.
Inference happens every time anyone uses the model, for as long as the product exists. Each request is small on its own, but there can be millions of them a day.
Training | Inference | |
|---|---|---|
What it does | Builds or adjusts the model | Runs the model to get an answer |
How often | Once, or now and then | Every time someone uses the product |
What matters most | Total computing power | Speed and cost per request |
Who pays | The team building the model | Whoever runs the product, for as long as it runs |
Most companies building AI products never train a model from scratch. They use a model someone else trained, through an API or as open weights, and pay for inference.
How does inference work?
What happens depends on the kind of model, but language models are the most common example.
1. The input becomes tokens. Your text is split into small pieces called tokens. A token is usually part of a word: "inference" might be one or two tokens, and a page of text is roughly 500 to 700. Images and audio are turned into numbers in a similar way.
2. The model reads the whole input. This first pass is called prefill. The model processes every input token at once and builds up a working memory of the conversation, called the KV cache.
3. The model writes the answer one token at a time. This is called decoding. The model predicts the most likely next token, adds it to the answer, and repeats. Each new token is based on everything before it, and this continues until the answer is finished. This is why chatbot answers appear word by word: they are streamed back as each token is produced.
4. The tokens become output. The tokens are turned back into text, or into an image, audio, or a structured result your application can use.
Image models work differently (many start from noise and refine it over a number of steps), and so do speech and classification models, but the idea is the same: an input goes in, the model runs on a processor, and an output comes back.
What are tokens, and why do they matter?
Tokens are the unit almost everything in inference is measured in. Model APIs charge per token, usually with separate prices for input and output tokens. Speed is measured in tokens per second. Limits on how much a model can read at once, the context window, are measured in tokens.
Output tokens are usually the expensive ones, because they are produced one at a time. A long answer costs more and takes longer than a short one, and a model that reasons step by step before answering can produce many more tokens than the answer you finally see.
Where does inference run?
Inference runs on whatever hardware the model is deployed to. The main options:
A model provider's cloud. You call an API from a provider such as OpenAI, Anthropic, or Google, and they run the model. You never see the hardware.
GPUs you rent in the cloud. You run a model yourself on GPUs from a cloud provider or a GPU host.
Your own data center. You run the model on hardware you own, on premises.
On the device. Small models can run on a phone, a laptop, or a machine in a factory, without sending data anywhere. This is often called edge inference.
Most production inference runs on GPUs, because they can do the huge number of calculations a model needs in parallel. Other chips are built for it too, such as Google's TPUs and specialized inference accelerators, and very small models can run on ordinary CPUs.
How is inference performance measured?
A few numbers come up again and again:
Latency: how long one request takes from start to finish.
Time to first token (TTFT): how long before the first word of an answer appears. This is what makes a chatbot feel fast or slow.
Tokens per second: how quickly the rest of the answer is produced once it starts.
Throughput: how many requests, or tokens, the whole system can handle per second. This decides how many users one GPU can serve.
Cost per request or per million tokens: what each answer actually costs you.
Utilization: how much of the time your GPUs are doing useful work. An idle GPU still costs money.
These pull against each other. Serving more users on the same GPU raises throughput and lowers cost, but can make each request a little slower. Good setups pick the balance that fits the product: a voice assistant needs a fast first response, while an overnight batch job only cares about cost.
Real-time and batch inference
Real-time (online) inference answers a user who is waiting: a chatbot, a search box, a voice agent. Speed matters most.
Batch (offline) inference processes a large pile of work with no one waiting: tagging a million product photos, summarizing last month's support tickets. Cost matters most, and the work can run when GPUs are cheaper or idle.
Why is everyone talking about inference now?
For the last few years the headlines were about training: bigger models, bigger clusters, bigger budgets. That is changing. At NVIDIA's GTC keynote in March 2026, Jensen Huang put it simply: "Inference is your workloads and tokens are your new commodity."
Three things are behind the shift:
AI is in everyday products. Each use of an AI feature is an inference request, and there are now billions of them a day.
Models think before they answer. Reasoning models work through a problem step by step before replying, which can mean many more tokens, and much more computing, per answer.
Agents make many calls. An AI agent that plans, searches, and checks its own work may call a model dozens of times to finish one task.
Training is a cost you plan for. Inference grows with your usage, every month.
What drives the cost of inference?
A few things decide what each request costs:
The size of the model. Bigger models need more computing and more GPU memory for every token.
How many tokens go in and come out. Long prompts, long answers, and step-by-step reasoning all add up.
The hardware. Which GPU you run on, what you pay for it, and how much of the time it sits idle.
How well it is set up. The same model can run several times faster on the same GPU depending on how it is configured.
How you pay. Per token through an API, or per hour for GPUs you run yourself.
That fourth point is the one most teams never get to. A model is usually set up once at launch and left alone, so the setup that made sense on day one is still running when traffic is ten times higher.
How do you make inference faster and cheaper?
This work is usually called inference optimization. The most common techniques, in plain terms:
Batching. Handling many requests together on the same GPU instead of one at a time. Continuous batching adds new requests as soon as there is room, rather than waiting for a whole group to finish. It is often the single biggest gain in throughput.
Quantization. Storing the model's numbers with less precision, for example 8 or 4 bits instead of 16. The model gets smaller and faster, usually with a small loss in quality that has to be tested.
Caching. Reusing work instead of repeating it. The KV cache keeps the model from rereading the whole conversation for every new token, and prompt caching reuses the work for text that repeats across requests, like a long system prompt.
Speculative decoding. A small, fast model drafts several tokens ahead and the large model checks them in one step, so the answer comes out faster without changing it.
Smaller, specialized models. A smaller model trained for one narrow task can often match a large general model on that task, at a fraction of the cost. This is often done through fine-tuning or distillation, where a small model learns from a larger one.
Splitting large models across GPUs. Models too big for one GPU are divided across several, which also needs tuning to avoid wasting time passing data between them.
Choosing the right hardware. Matching the GPU type and number of GPUs to the model and the traffic, and scaling up and down as demand changes.
Teams usually run models with serving software built for this, such as vLLM, SGLang, TensorRT-LLM, or llama.cpp, which handle much of the batching and caching. Getting the most out of them still means tuning for each model, each GPU, and each traffic pattern, and retesting quality after every change.
Should you use an API or run your own model?
Most companies pay for inference in one of two ways.
A per-call API. You send each request to a model provider and pay for every call or every token.
Good: the fastest way to start, no hardware to manage, access to the largest models.
Watch out for: the bill grows exactly as fast as your usage, prices and models can change under you, and your data leaves your company with every request.
Usually the right choice for prototypes, low or unpredictable volume, and tasks that need the largest models.
Running the model yourself. You run a model you control, usually an open-weight model, on GPUs you rent or own, and pay for the hardware rather than per request.
Good: the cost stays flat as volume grows, so each request gets cheaper the more you use it; your data stays where you put it; you control the model and when it changes.
Watch out for: someone has to deploy the model, tune it, scale it, and get paged at three in the morning when it breaks. Idle GPUs still cost money.
Usually the right choice for steady, high volume on one or a few tasks, and for data that has to stay in your own environment.
For one of our customers, Roomates, the difference was large. They were paying 4 cents per image on a per-image API. After moving their model onto a single GPU for a fixed $400 a month, the cost per image falls to $0.0005 at the GPU's projected capacity, about 99% less, and each image takes 3 seconds instead of 12. You can read the full case study here.
Frequently asked questions
Is inference the same as running a model? Yes. Inference, serving, and running a model in production all describe the same thing: using a trained model to answer requests.
What is an inference endpoint? The address your application sends requests to. Behind it, the model runs on one or more GPUs and sends the answer back.
Why is inference so expensive? Because it never stops. Training is paid for once, but every user request costs something, and large models need expensive GPUs to answer quickly. Costs grow with usage, especially for long answers and reasoning models.
What is the difference between latency and throughput? Latency is how long one user waits. Throughput is how much work the whole system gets through. Serving more users at once improves throughput but can raise latency, so the right balance depends on the product.
Can inference run without a GPU? Small models can run on CPUs, phones, and laptops. Most large models need GPUs or other AI accelerators to answer at a usable speed.
Does inference use my data to train the model? Not by itself. Inference only runs the model. Whether your prompts are stored or used for training depends on the provider's terms, which is one reason some companies run models in their own environment.
What is inference optimization? Making a model answer faster or more cheaply without hurting quality, using techniques like batching, quantization, caching, and smaller specialized models.
Inference at Sparset
Inference is the work we do at Sparset. Our agents take a company's model, make it cheaper and faster to run, deploy it on our GPUs or the company's own, and keep it running and improving every day, so the company does not need an infrastructure team to do it. If you want to know what your inference should cost, book a call and we will take a look.