Streaming

Stream Responses events

import OpenAI from "openai";
const client = new OpenAI({
  baseURL: "https://www.cheapinference.dev/v1",
  apiKey: process.env.CHEAPINFERENCE_API_KEY,
});

const stream = await client.responses.create({
  model: "gpt-5.4-mini", input: "Write a short welcome.",
  max_output_tokens: 1024, stream: true, store: false,
});
for await (const event of stream) {
  if (event.type === "response.output_text.delta") process.stdout.write(event.delta);
  if (event.type === "response.completed") console.log("\nUsage", event.response.usage);
  if (event.type === "response.failed") console.error("Generation failed", event.response.error);
  if (event.type === "response.incomplete") console.error("Incomplete", event.response.incomplete_details);
}

Other events announce output items, function-call arguments, reasoning summaries, and content boundaries. Handle event types deliberately rather than assuming every event contains text.

Stream Chat Completions

import OpenAI from "openai";
const client = new OpenAI({
  baseURL: "https://www.cheapinference.dev/v1",
  apiKey: process.env.CHEAPINFERENCE_API_KEY,
});

const stream = await client.chat.completions.create({
  model: "gpt-5.4-mini",
  messages: [{ role: "user", content: "Write a short welcome." }],
  max_completion_tokens: 1024, stream: true,
});
for await (const chunk of stream) {
  const text = chunk.choices[0]?.delta?.content;
  if (text) process.stdout.write(text);
  if (chunk.usage) console.log("\nUsage", chunk.usage);
}

We always request the terminal usage chunk. It can contain an empty choices array. Tool-call argument deltas must be accumulated before parsing their JSON.

Raw HTTP and SSE framing

Check the HTTP status before reading SSE. Each event ends with a blank line, not necessarily a network chunk boundary. Buffer partial frames, support CRLF, combine repeated data lines, ignore comment lines beginning with :, and treat [DONE] as a sentinel when present. Prefer the official SDK or a standards-compliant SSE parser.

Errors before the stream starts use normal HTTP status codes. Errors after headers are committed can be an SSE error event or a failed terminal response while HTTP remains 200. Handle these separately from network errors.

Disconnects, retries, and billing

If your client disconnects, upstream work may continue. The metering proxy drains the provider stream to capture the final usage; completed work remains billable. Failed terminal generations with usage are recorded as failures with their observed usage. Missing reports are marked for review.

Do not automatically replay a stream that already produced output. The service does not deduplicate inference retries. Browser-session routes never sit in the API authentication path.