504 gateway timeout on an LLM API: what actually timed out, and why go2llm answers 502 instead

Q: I'm seeing a real 504 with no JSON body at all. Is that go2llm?

Almost certainly not directly. A body-less, non-JSON 504 points at something between your code and go2llm's edge — your own proxy, load balancer, or client library — rather than the gateway's own error-handling path, which always answers in the JSON envelope shown above.

Q: Is a timeout the same thing as the model being overloaded?

Not necessarily. Overload usually surfaces as a fast 429 or 502/503 — the provider answers quickly to say it can't take the request. A timeout means nothing came back in time at all, which can also just mean a large, slow generation that was always going to take a while.

504 Gateway Timeout is a generic reverse-proxy status: something sitting in front of the real server gave up waiting for an answer and told you so itself. Model providers don't usually emit a 504 as their own considered response — a model either answers, or an upstream 5xx comes back, or nothing comes back before something in the chain decides to stop waiting. A 504 almost always means the "something" was a proxy, load balancer, or CDN between you and the API, not the model provider's own code.

go2llm runs its own internal deadline on every upstream call — 180 seconds on most routes, 60 seconds on a few. What the gateway does when that ceiling fires depends on whether its provider router is switched on. With the router on, the deadline expiring returns 502 with error.code: "timeout", and if the provider itself hands back a real 504 — its own edge genuinely timing out — the router relays that status straight through instead, as a literal 504 with error.code: "upstream_http_504". With the router off, both cases collapse into the same generic 502 upstream_unavailable, message "all safe upstream attempts failed" — there is no error.code: "timeout" on that path, because the older code retries across other routes first and reports one combined failure once they're all exhausted. Check which mode is live with no key required — curl -s https://go2llm.tech/status and the providerRouterV1 field; it's true on go2llm.tech today, but that's a live setting, not a permanent default, so verify it rather than trusting this page to stay current.

Written 2026-10-02.

What it looks like

A 504 from a reverse proxy or load balancer typically isn't even a JSON body — it's often plain text or a stock HTML error page, because it was generated by infrastructure that has never heard of go2llm's error envelope:

504 Gateway Time-out
The server didn't respond in time.

go2llm's own deadline elapsing, when the provider router is on, always comes back as JSON, with status 502 and error.code: "timeout":

{
  "error": {
    "type": "upstream_error",
    "code": "timeout",
    "message": "The upstream model is temporarily unavailable. Retry the request (request_id: gw_5e0c92a1d7b3486fa4e18c05b2d7f639).",
    "request_id": "gw_5e0c92a1d7b3486fa4e18c05b2d7f639"
  }
}

With the router off, the same deadline elapsing looks different — there's no per-cause code, just the shared upstream-failure envelope:

{
  "error": {
    "type": "upstream_unavailable",
    "code": "upstream_unavailable",
    "message": "all safe upstream attempts failed",
    "request_id": "gw_5e0c92a1d7b3486fa4e18c05b2d7f639"
  }
}

A literal 504 with that same kind of envelope is also possible, when the provider router is on: it makes one attempt per request and relays whatever status the provider itself answered with, so a genuine timeout on the provider's own edge passes straight through as 504, with error.code: "upstream_http_504". This is rare — providers themselves seldom emit a considered 504 — but it does happen, and it's a different situation from go2llm's own deadline above. With the router off, this case folds into the same generic 502 upstream_unavailable as go2llm's own deadline — the two aren't distinguishable by code on that path.

If you're seeing a literal 504 from go2llm.tech with no JSON error object at all — a bare status and nothing else — *that* response almost certainly didn't come from the gateway's own code. It came from something between your code and the gateway's edge: a corporate proxy, a load balancer in your own deployment, or a client library's own timeout wrapper labeling a hang as "504" in its own logs. The JSON envelope is what tells the two situations apart.

What causes it, most common first

1. Your own HTTP client timeout is shorter than the model actually needs

By far the most common cause. A long completion from a large model can run for tens of seconds; a client configured with a 10- or 20-second timeout cuts the connection first and reports it as a timeout — sometimes surfaced by the client library itself as a "504"-flavored error even though nothing labeled it that on the wire. Nothing was wrong with the request; the wait was just longer than the client was configured to tolerate.

2. A proxy or load balancer you put in front of your own call to go2llm

If your code talks to go2llm through your own API gateway, ingress, or corporate proxy, that layer has its own timeout — and it's often shorter than go2llm's own 180-second (or 60-second) internal deadline. In that case, your infrastructure gives up and returns a real 504 several seconds or more before go2llm would have answered on its own, successfully or not.

3. go2llm's own upstream deadline elapsing

go2llm gives an upstream call up to 180 seconds on most routes, 60 seconds on some. When that ceiling is reached before the provider answers at all, the gateway returns 502 — with error.code: "timeout" when the provider router is on, or the shared upstream_unavailable code when it's off — without indicating which of the two ceilings applied either way; a deliberate design choice, not a bug, and distinct from the upstream_http_5xx code a real provider response gets relayed under when the router is on (see 500 internal server error).

4. A long, un-streamed generation on a slow model

Requesting a large max_tokens from a slower model without streaming means waiting for the entire completion to finish before a single byte comes back. This isn't a malfunction — it's simply how long the full generation takes — but it looks identical to a hang from the client's point of view until either the model finishes or some timeout fires first.

5. The vendor's own edge, when calling a provider directly

Providers commonly sit behind their own CDN or edge network, which can time out a slow origin response on its own terms before the origin API itself would have. This applies to calling a vendor's API directly, not to calls through go2llm, and it's worth ruling out separately if you're comparing behavior between the two.

Testing which one you're looking at

Check which mode is live first — curl -s https://go2llm.tech/status and the providerRouterV1 field, since that decides which error.code to expect below. Then measure the actual wall-clock time to failure with curl's own timing, not your application's timeout:

curl -s -o /dev/null -w "total time: %{time_total}s\nhttp code: %{http_code}\n" \
  --max-time 200 \
  https://go2llm.tech/v1/chat/completions \
  -H "Authorization: Bearer $GO2LLM_API_KEY" \
  -H "content-type: application/json" \
  -d @request-body.json

If the cutoff lands close to 60 or 180 seconds and comes back as JSON — with error.code: "timeout" when the router is on, or the shared upstream_unavailable code when it's off — that's go2llm's own internal deadline, not the provider's. If it comes back sooner with a different upstream_http_5xx code, including 504 (router on only — with the router off this case also folds into upstream_unavailable), that's the provider's own response, relayed as-is. If it cuts off at some other round number — 10, 15, 30 seconds — with no JSON body at all, look at your own HTTP client's timeout setting and any proxy sitting between your code and go2llm.tech.

Fixing it

Raise your client's own read timeout. Set it comfortably above 180 seconds for non-streaming calls, so your own client isn't the one giving up first:

import httpx

client = httpx.Client(timeout=httpx.Timeout(200.0, connect=10.0))

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.GO2LLM_API_KEY,
  baseURL: "https://go2llm.tech/v1",
  timeout: 200_000, // milliseconds — comfortably above go2llm's own 180s ceiling
});

Switch to streaming for anything that might run long. Streaming delivers the first tokens quickly instead of making the client wait for the entire completion, which sidesteps most naive timeout configurations entirely:

curl https://go2llm.tech/v1/chat/completions \
  -H "Authorization: Bearer $GO2LLM_API_KEY" \
  -H "content-type: application/json" \
  -d '{"model":"claude-sonnet-5","stream":true,"max_tokens":1024,"messages":[{"role":"user","content":"Write a short paragraph"}]}'

If you run your own proxy in front of go2llm, raise its timeout too, independently of your HTTP client's own setting — the two are configured in different places and one being generous doesn't help if the other still cuts the connection first.

Cap max_tokens to what a latency-sensitive path actually needs. A smaller reply finishes sooner regardless of which timeout would otherwise have fired.

Avoiding it later

Prefer streaming by default for anything that isn't guaranteed to be short — it turns "silence for two minutes, then an answer or a failure" into steady partial progress, which is both a better experience and much harder to time out on by accident. In your own retry logic, treat "no response after roughly 185 seconds" the same as a 502: something upstream failed to answer in time, and it's safe to retry a plain text request on that basis. Recognize the failure by status and by code family — >= 500, plus any error.code equal to timeout or upstream_unavailable, or starting with upstream_http_ — rather than one specific code, since which one you get depends on whether the provider router is on for that request. Don't apply the same logic to a stream that already delivered at least one valid event — see 502 upstream_unavailable and 503 for why a mid-stream retry risks a duplicate answer and a duplicate charge.

A request that never got an answer was never billed. The cost estimate reserved before the call is released on any failure, timeout included, and only a response that actually completes settles a real cost in usage.cost_usd.

Related errors

502 upstream_unavailable and 503 — the general upstream-failure page and a full retry loop with jitter.
429 rate_limit_error — a different reason a request doesn't complete quickly.
The full error reference.

FAQ

Does go2llm ever send a literal 504? Yes, when the provider itself answers with one and the provider router is on for that request — rare, since providers seldom emit a considered 504, but the router relays it verbatim, error.code: "upstream_http_504" included. Check providerRouterV1 at /status to know if that applies to you. go2llm's own upstream deadline firing is a different thing: with the router on it's a 502 with error.code: "timeout"; with the router off, both that and a provider's literal 504 collapse into the same generic 502 upstream_unavailable. Either way, go2llm's own deadline never shows up as a literal 504.

What are go2llm's actual timeout numbers? 180 seconds on most routes, 60 seconds on a smaller set of routes. Neither figure is broken down further per model in the public docs; if a request is time-sensitive, design for the shorter number to be safe.

I'm seeing a real 504 with no JSON body at all. Is that go2llm? Almost certainly not directly. A body-less, non-JSON 504 points at something between your code and go2llm's edge — your own proxy, load balancer, or client library — rather than the gateway's own error-handling path, which always answers in the JSON envelope shown above.

Is a timeout the same thing as the model being overloaded? Not necessarily. Overload usually surfaces as a fast 429 or 502/503 — the provider answers quickly to say it can't take the request. A timeout means nothing came back in time at all, which can also just mean a large, slow generation that was always going to take a while.

Get a key and make your first request in 3 minutes → go2llm.tech. Per-model prices are on the pricing page, setup steps are in the docs.