Predicted Outputs

Regenerate mostly unchanged text

Predicted Outputs can reduce latency when a compatible model regenerates a document with small edits. This is a Chat Completions feature. Support depends on the model; the provider may reject incompatible combinations.

import OpenAI from "openai";
const client = new OpenAI({
  baseURL: "https://www.cheapinference.dev/v1",
  apiKey: process.env.CHEAPINFERENCE_API_KEY,
});

const existing = "The app opens at 9am and closes at 5pm.";
const result = await client.chat.completions.create({
  model: "gpt-4.1",
  messages: [{ role: "user", content: `Change the opening time to 10am: ${existing}` }],
  prediction: { type: "content", content: existing },
  max_completion_tokens: 256,
});
console.log(result.choices[0]?.message.content);
console.log(result.usage?.completion_tokens_details);

Accepted and rejected prediction tokens

The response may include accepted_prediction_tokens and rejected_prediction_tokens. Rejected predictions are still billable. They are included in the provider’s output-token accounting and must not be added a second time.

The initial hold includes an extra conservative allowance for prediction content, so enabling a prediction can require more available credit. Final settlement uses reported usage. For missing-usage anomalies, the estimated reservation remains subject to review as documented in billing.