How do you call an LLM API?

What happens in one API call

An LLM API is a web service. Your program sends text, the provider's servers run the model, and the result comes back as JSON. There is no memory between calls: every request must include the whole conversation that the model should see.

You need three things: an account with an API key, the URL of the endpoint, and a model ID from the provider's model list. In the examples below, MODEL_ID is a placeholder. Model names change often, so copy the current ID from the provider's documentation instead of from a blog post.

A minimal request in curl, Python and JavaScript

The chat completions format, first published by OpenAI, is the most widely copied request shape. Many other providers and local servers accept it, sometimes with only a different base URL. Here is the smallest useful request:

curl https://api.openai.com/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -d '{
    "model": "MODEL_ID",
    "messages": [
      {"role": "system", "content": "You answer in one short paragraph."},
      {"role": "user", "content": "What is a context window?"}
    ]
  }'

The messages list holds the conversation. Each message has a role and content. The system message sets rules for the reply (OpenAI's newer models also accept the name developer for this role). user messages are the requests, and assistant messages are earlier replies, which you send back to continue a conversation. A shortened response looks like this:

{
  "id": "chatcmpl-...",
  "object": "chat.completion",
  "model": "MODEL_ID",
  "choices": [
    {
      "index": 0,
      "message": {"role": "assistant", "content": "A context window is ..."},
      "finish_reason": "stop"
    }
  ],
  "usage": {"prompt_tokens": 27, "completion_tokens": 64, "total_tokens": 91}
}

The text is in choices[0].message.content. The usage object counts the tokens you are billed for. finish_reason tells you why the model stopped: stop means it finished, while length means it hit your token limit and the answer is cut off.

In Python, the same call with a timeout and retries looks like this. It uses the common requests library. Official SDKs from each provider do the same work and add retries for you.

import os, random, time
import requests

URL = "https://api.openai.com/v1/chat/completions"
API_KEY = os.environ["OPENAI_API_KEY"]   # read from the server environment

def chat(messages, model="MODEL_ID", max_attempts=5):
    body = {"model": model, "messages": messages,
            "max_completion_tokens": 500}   # many compatible APIs: "max_tokens"
    headers = {"Authorization": f"Bearer {API_KEY}"}
    for attempt in range(max_attempts):
        last = attempt == max_attempts - 1
        try:
            r = requests.post(URL, headers=headers, json=body, timeout=60)
        except (requests.Timeout, requests.ConnectionError):
            if last:
                raise
            time.sleep(2 ** attempt + random.random())   # network error: retry
            continue
        if r.status_code == 429 or r.status_code >= 500:
            if last:
                break
            backoff = 2 ** attempt + random.random()      # exponential + jitter
            try:
                wait = max(float(r.headers.get("Retry-After", 0)), backoff)
            except ValueError:
                wait = backoff
            time.sleep(wait)
            continue
        r.raise_for_status()                # 400, 401, 403: fix the request, do not retry
        data = r.json()
        return data["choices"][0]["message"]["content"], data["usage"]
    raise RuntimeError("API still failing after retries")

text, usage = chat([{"role": "user", "content": "What is a context window?"}])
print(text)
print(usage["prompt_tokens"], "in /", usage["completion_tokens"], "out")

Providers differ: OpenAI-compatible, Anthropic and Gemini

The chat completions shape is common but not universal. Anthropic's Messages API and Google's Gemini API have their own native formats. Both providers also publish compatibility options, but their native APIs expose all features first. The main differences:

Gemini offers an OpenAI-compatible base URL, and many other providers, including DeepSeek and several Chinese and open-model hosts, accept the chat completions format. Compatibility is usually partial: advanced features such as structured output, tool calling or image input may differ. Read each provider's compatibility page before switching with only a URL change.

Keep the API key on the server

Anyone who has your API key can spend your money. Never put it in browser JavaScript, a mobile app, a public repository or a front-end environment file that gets bundled into the page. All of these can be read by users.

1. Store the key in an environment variable or a secret manager on your server.
2. Have your web or mobile app call your own backend endpoint.
3. On the backend, check the user (login, rate limit per user, input length), then call the LLM API with the key.
4. Return only the result to the client.
5. Set a monthly spending limit in the provider's dashboard, use separate keys per project, and rotate a key immediately if it leaks.

Streaming: show the answer as it is written

A long answer can take many seconds. With "stream": true, the server sends small pieces as server-sent events, so your interface can show text as it appears. Each event line starts with data: and holds a chunk with a delta. The OpenAI format ends with data: [DONE]. This example runs on a Node.js server:

// Node.js 18+ (server side). Prints the reply as it arrives.
const res = await fetch("https://api.openai.com/v1/chat/completions", {
  method: "POST",
  headers: {
    "Content-Type": "application/json",
    Authorization: `Bearer ${process.env.OPENAI_API_KEY}`,
  },
  body: JSON.stringify({
    model: "MODEL_ID",
    stream: true,
    messages: [{ role: "user", content: "Write a haiku about rain." }],
  }),
});
if (!res.ok) throw new Error(`HTTP ${res.status}: ${await res.text()}`);

const decoder = new TextDecoder();
let buffer = "";
for await (const chunk of res.body) {
  buffer += decoder.decode(chunk, { stream: true });
  const lines = buffer.split("\n");
  buffer = lines.pop();                       // keep a partial line for later
  for (const line of lines) {
    if (!line.startsWith("data: ")) continue;
    const data = line.slice(6);
    if (data === "[DONE]") continue;          // end of stream
    const delta = JSON.parse(data).choices[0]?.delta?.content;
    if (delta) process.stdout.write(delta);
  }
}

To show the stream in a browser, forward the chunks from your server to the page. Do not call the provider from the browser with your key. In the OpenAI format, usage is not included in streamed chunks unless you set stream_options with include_usage, so turn that on if you need token counts for billing.

Getting JSON instead of prose

When code will read the answer, ask for JSON. There are two levels. JSON mode ("response_format": {"type": "json_object"}) promises valid JSON, but not which fields. Structured output with a JSON Schema makes the model follow your exact fields and types, on models that support it:

{
  "model": "MODEL_ID",
  "messages": [
    {"role": "user", "content": "Extract the total and currency: 'Amount due: 1.250,00 EUR'"}
  ],
  "response_format": {
    "type": "json_schema",
    "json_schema": {
      "name": "invoice_total",
      "strict": true,
      "schema": {
        "type": "object",
        "properties": {
          "total": {"type": "number"},
          "currency": {"type": "string"}
        },
        "required": ["total", "currency"],
        "additionalProperties": false
      }
    }
  }
}

Even with a schema, validate the parsed result in your code. The values can still be wrong, for example a misread number, even when the shape is right. Anthropic and Gemini support schemas and tool definitions in their own formats. When you debug long JSON replies, a formatter such as cv.cm's json tool makes nested output easier to read.

Rate limits, retries and errors

Providers limit both requests per minute (RPM) and tokens per minute (TPM), and whichever you hit first applies. When you exceed a limit, you get HTTP status 429. Handle errors by type:

Retry with exponential backoff: wait about 1 second, then 2, 4, 8, plus a small random delay (jitter) so many clients do not retry at the same moment. If the response has a Retry-After header, wait at least that long. Cap the number of attempts. Failed requests can also count toward your limit, so retrying instantly in a tight loop makes things worse. If you use an official SDK, remember it already retries some errors.

Cost is counted in tokens

You pay for input tokens (everything you send, including the system prompt and the whole conversation history) and output tokens (what the model writes). Output tokens are usually priced higher than input tokens. The formula is simple:

cost = input_tokens  / 1,000,000 * input_price_per_million
     + output_tokens / 1,000,000 * output_price_per_million

Example with made-up prices (2.00 in, 8.00 out per million tokens):
  27 input + 64 output tokens
  = 0.000054 + 0.000512 = 0.000566 per call
  x 100,000 calls a month = about 56.60

• Read real counts from usage in each response and log them. Do not estimate from word counts.
• Cap the length of each reply: max_completion_tokens on OpenAI (the older max_tokens is deprecated there), max_tokens on Anthropic and many compatible APIs.
• Chat history grows with every turn, so the 20th message costs more than the first. Trim or summarize old turns.
• Use a smaller model for simple tasks such as classification, and a larger one only where quality needs it.
• Some providers discount repeated prompt prefixes (prompt caching) or offer cheaper batch processing for jobs that can wait.