Every request to Topo picks a node before it picks a token. The router has about 15 ms to choose among hundreds of candidates in 40+ regions, and that choice decides most of the latency you see.
How a node gets picked
- 01Filter to nodes that serve the model and already hold its weights in memory.
- 02Score each one on live queue depth, network distance and recent verification record.
- 03Send the request to the best node and keep the runner-up as a hedge.
The hedge matters more than the score. If the first node has not streamed a token inside its expected window, the router sends the same request to the runner-up and keeps whichever answers first. You are billed once, for the stream you receive.
x-topo-route
json
{"node": "us-east-4","gpu": "H100","hops": 2,"queue_ms": 11,"ttft_ms": 182,"hedged": false}From 240 ms to 182 ms
Most of the win came from keeping weights resident on more nodes and from scoring on live queue depth instead of a five second average. Region pinning on Max plans uses the same scorer with a smaller candidate set, so pinned traffic gets the same treatment.