Running the
model.
Training builds the brain once. Inference is what happens every single time you actually use it — the model generating an answer, one token at a time. It's where AI meets the real world, and where the bill comes due.
"Inference" is the industry word for using a trained model. You send it text, it predicts the next token, appends it, predicts again, and keeps going until it decides it's finished. That's it — the same next-token trick from training, except now the weights are frozen and it's generating an answer instead of learning.
One token at a time
Ever notice how ChatGPT or Claude types its answer out word by word instead of dropping it all at once? That's not a stylistic choice — it's literally how the model works. Each token has to be generated before the next one can be, because every new token depends on all the tokens before it. The model can't know word ten until it has committed to word nine. Streaming just shows you that process live.
Speed is measured in tokens per second (tok/s). A snappy assistant runs ~30–100+ tok/s. Bigger models are smarter but slower; smaller models are faster but less capable. Choosing the right size for the job is half the engineering.
You pay by the token, forever
This is the fact that reshapes every AI budget: training is spent once; inference is spent on every request, forever. Send 1,000 tokens in and get 500 back, and you've paid for 1,500 tokens of compute — and you'll pay again on the next message, and the next. At scale, inference is almost always the dominant cost of running an AI product, not training.
It's also why "just use the biggest model for everything" is usually the wrong move. A large frontier model might cost 20–50× more per token than a smaller one that handles the task fine. Good AI engineering is largely about routing — sending easy work to cheap fast models and reserving the expensive ones for the hard, high-judgment steps.
Every token you send gets re-read
Because the model considers its whole context window to produce each answer, a long conversation or a big pasted document isn't free — all of it is processed on every turn. That's why costs creep up in long chats, and why techniques like retrieval (fetching only the few relevant snippets instead of dumping everything in) and caching (reusing work on repeated context) matter so much in a real system.
On a chip, somewhere
Every token you just watched stream out was computed on physical silicon — a GPU, FPGA, or ASIC — sitting either in a cloud data center or, in our case, on a rack in the office. Whether that hardware is rented from a frontier lab or owned outright changes the economics, the privacy, and the control completely. Those two paths are the last piece of the picture.
Common questions
What is inference in AI?
Inference is running a trained model to generate an answer. You send it text, it predicts the next token, appends it, and repeats until it's done. It's the same next-token prediction as training, except the weights are frozen and the model is producing output instead of learning.
Why does AI generate text one word at a time?
Each token depends on all the tokens before it, so the model literally cannot produce word ten until it has committed to word nine. That sequential dependency is why chat assistants stream their answers out token by token instead of all at once.
Why is AI inference expensive?
Training is paid once, but inference is paid on every request — forever. Every token you send and every token generated costs compute, and long conversations or big documents get re-read on each turn. At scale, inference is usually the dominant cost of running an AI product.
What are tokens per second (tok/s)?
Tokens per second measures how fast a model generates output. A snappy assistant runs roughly 30–100+ tok/s. Larger models are smarter but slower; smaller models are faster but less capable, so matching model size to the task is key.