A fallback is a product decision.
Design retry and fallback behavior around what your application can safely repeat.
PrismHub editorial·4 min readClassify the failure first
A rate limit, timeout, invalid request, and authentication failure need different responses. Retrying a malformed request with another provider usually adds delay without fixing the problem.
Preserve the capability contract
A fallback must support the inputs, tools, and output format your application needs. Check these requirements before evaluating speed or price. Similar model names do not imply equivalent behavior.
Set a finite retry budget
Use a request deadline, bounded attempts, and exponential backoff with jitter where appropriate. Retry-After headers can help with rate limits. Avoid multiplying retries independently at every layer of your stack.
Keep side effects out of automatic retries
A model can suggest a tool call more than once. Use application-level idempotency and confirmation for actions such as sending a message or charging a payment. Record which model and attempt produced the accepted output.