Short pieces on how the serving stack behaves, written when a number surprised us.
A cached token skips prefill entirely, which is the expensive half of serving a long prompt. It costs us close to nothing, so charging the full input rate for it would be charging for work we did not do.
Publishing the rate in /v1/models also lets a client price a prompt before sending it. At a 64% hit rate on deepseek4-flash, a million input tokens costs $0.0837 rather than $0.22, which you can compute from two fields in the feed.
All three of the large open models now serve a 262,144-token window with up to 131,072 tokens of output. The window is not the constraint most people hit first.
Prefill is. A 200k-token prompt spends most of its latency before the first token appears, which is exactly the case prefix caching removes on the second call. If you are looping over one long document, order your messages so the document comes first.
There is no automatic routing here and no scoring layer picking a model for you. You name a model and that model answers.
A multi-model API is a short list with the prices and capabilities of each model published as data. Everything else is a preference you express in your own code.