Cookbook Recipe
Batch ProcessingRun batch work on Flex with automatic fallback
A batch program that sends every item on the Flex tier, resends an item on the standard tier when Flex has no capacity, and waits only when the response gives a delay, so the batch keeps moving.
What you'll build
A Python or Node.js program that classifies a list of short reviews with a small worker pool, sending each request to /v1/chat/completions with "service_tier": "flex". Flex is the tier for work that does not need an immediate answer (see Service tiers); when an item gets no Flex capacity, the program resends it at once on the standard tier, so the item still gets an answer when the standard tier can take it. At the end it prints the tier that served each item and a summary of served tiers and error codes.
Before you start
- A DEVUP AI account and an API key (
sk-devup-...) stored asDEVUP_API_KEY. See Authentication. - Python 3.10 or later with the
openaipackage, or Node.js 22.12 or later with the officialdevupaiSDK. - The program uses
deepseek-ai/DeepSeek-V4-Flash. To use another model, check that its model page lists Flex among its service tiers; a model without Flex processes the request on the standard tier.
pip install openaiTo try a single Flex request first:
curl https://api.devupai.com/v1/chat/completions \
--max-time 600 \
-H "Authorization: Bearer $DEVUP_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "deepseek-ai/DeepSeek-V4-Flash",
"service_tier": "flex",
"messages": [{ "role": "user", "content": "Hello!" }]
}'The complete program
Save it as flex_batch.py or flex_batch.mjs and run it with DEVUP_API_KEY set. The Node.js program uses the official devupai SDK. The SDK's own retries are turned off so this code decides when to retry and when to fall back.
"""Run a batch on the Flex tier and fall back to the standard tier when Flex has no capacity."""
import math
import os
import sys
import time
from collections import Counter
from concurrent.futures import ThreadPoolExecutor
import openai
from openai import OpenAI
MODEL = "deepseek-ai/DeepSeek-V4-Flash"
WORKERS = 4 # requests in flight at the same time
MAX_WAIT_PER_RETRY = 60.0 # seconds, cap for one wait
MAX_TOTAL_WAIT = 300.0 # seconds, cap for all waits of one item
STANDARD_CAPACITY_RETRIES = 3 # retries of model_at_capacity on the standard tier
# On a Flex request, these codes mean Flex has no capacity now: resend on the standard tier.
FALLBACK_CODES = {"flex_capacity_unavailable", "model_at_capacity"}
# The account's own limits: wait, then retry on the same tier.
ACCOUNT_LIMIT_CODES = {"rate_limit_exceeded", "tpm_limit_exceeded", "concurrency_limit_exceeded"}
REVIEWS = [
"The delivery was fast and the product works as described.",
"The battery lasts half a day, much less than advertised.",
"It does the job. Nothing more, nothing less.",
"Support answered within an hour and solved my issue.",
"The screen cracked after one week of normal use.",
"Setup took five minutes, I would buy it again.",
]
def _parse_number(value):
"""A finite float from a header value, or None."""
if value is None:
return None
try:
number = float(value)
except ValueError:
return None
return number if math.isfinite(number) else None
def retry_delay_seconds(headers):
"""Seconds to wait before a retry: retry-after-ms first, then Retry-After.
A missing value or a value <= 0 counts as 1 second; one wait is capped at MAX_WAIT_PER_RETRY.
"""
ms = _parse_number(headers.get("retry-after-ms"))
seconds = ms / 1000 if ms is not None else _parse_number(headers.get("retry-after"))
if seconds is None or seconds <= 0:
seconds = 1.0
return min(seconds, MAX_WAIT_PER_RETRY)
def send(client, item, tier):
"""One chat completion for an item. tier is "flex" or "standard" (no service_tier)."""
request = {
"model": MODEL,
"messages": [
{
"role": "user",
"content": "Classify the sentiment of this review as positive, negative or neutral. "
"Reply with one word.\n\nReview: " + item["prompt"],
}
],
}
if tier == "flex":
request["service_tier"] = "flex"
return client.chat.completions.create(**request)
def process_item(client, item, sleep=time.sleep, now=time.monotonic):
"""Send one item on Flex; fall back to the standard tier or wait, depending on the error code."""
started = now()
tier = "flex"
waited = 0.0
capacity_retries = 0
while True:
try:
response = send(client, item, tier)
except openai.APIStatusError as e:
code = e.code or f"http_{e.status_code}"
request_id = e.response.headers.get("x-request-id")
# Its outcome is not confirmed yet: record it as unknown, with its request id.
if e.status_code == 409 and code == "request_outcome_pending":
return {"status": "unknown", "code": code, "request_id": request_id}
if e.status_code == 429 and tier == "flex" and code in FALLBACK_CODES:
tier = "standard" # resend at once, without service_tier
continue
can_wait = code in ACCOUNT_LIMIT_CODES or (
code == "model_at_capacity" and capacity_retries < STANDARD_CAPACITY_RETRIES
)
if e.status_code == 429 and can_wait:
delay = retry_delay_seconds(e.response.headers)
if waited + delay <= MAX_TOTAL_WAIT:
if code == "model_at_capacity":
capacity_retries += 1
waited += delay
sleep(delay)
continue
return {"status": "error", "code": code, "request_id": request_id}
# A timed-out request may have run: record it as unknown.
except openai.APITimeoutError:
return {"status": "unknown", "code": "timeout", "request_id": None}
except openai.APIConnectionError:
return {"status": "unknown", "code": "connection_error", "request_id": None}
return {
"status": "ok",
"tier": response.service_tier or "unreported",
"answer": response.choices[0].message.content or "",
"seconds": round(now() - started, 1),
}
def run_batch(client, items, results, workers=WORKERS, sleep=time.sleep, now=time.monotonic, resend_unknown=False):
"""Process every item without a finished result and store each outcome in results by item id.
Items with status "unknown" are sent again only with resend_unknown=True.
"""
skip = {"ok"} if resend_unknown else {"ok", "unknown"}
pending = [item for item in items if results.get(item["id"], {}).get("status") not in skip]
with ThreadPoolExecutor(max_workers=workers) as pool:
outcomes = pool.map(lambda item: process_item(client, item, sleep, now), pending)
for item, outcome in zip(pending, outcomes):
results[item["id"]] = outcome
return results
def summarize(results):
"""Counts per served tier, and errors by code."""
tiers = Counter({"flex": 0, "default": 0, "unreported": 0})
errors = Counter()
for outcome in results.values():
if outcome["status"] == "ok":
tiers[outcome["tier"]] += 1
else:
errors[outcome["code"]] += 1
return tiers, errors
def main():
api_key = os.environ.get("DEVUP_API_KEY")
if not api_key:
print("Set DEVUP_API_KEY.", file=sys.stderr)
sys.exit(1)
client = OpenAI(
api_key=api_key,
base_url="https://api.devupai.com/v1",
timeout=600,
max_retries=0,
)
items = [{"id": f"review-{n}", "prompt": text} for n, text in enumerate(REVIEWS, start=1)]
results = {} # load saved results here to skip finished items on a rerun
run_batch(client, items, results)
for item in items:
outcome = results[item["id"]]
if outcome["status"] == "ok":
print(f"{item['id']} -> {outcome['tier']} ({outcome['seconds']} s)")
else:
print(f"{item['id']} -> {outcome['status']} {outcome['code']} (x-request-id: {outcome['request_id']})")
tiers, errors = summarize(results)
print("Served tiers:", dict(tiers))
print("Errors:", dict(errors))
if __name__ == "__main__":
main()
Each line of output names an item and the tier that served it (flex, default or unreported), or the status (error or unknown), code and x-request-id of an item that has no answer. The last two lines count items per served tier and items without an answer per code.
How it works
1Give each request time for the wait and the reply
A Flex request can wait for capacity before processing starts (see Flex processing), and a non-streamed reply arrives once it is complete. The client timeout covers the Flex wait plus the reply. Service tiers asks for at least 300 seconds, which covers the wait; this recipe sets 600 seconds to leave extra room for long replies (timeout=600 in seconds in Python, timeout: 600_000 in milliseconds in Node.js). The program checks DEVUP_API_KEY before it builds the client and stops with exit code 1 when it is not set.
def main():
api_key = os.environ.get("DEVUP_API_KEY")
if not api_key:
print("Set DEVUP_API_KEY.", file=sys.stderr)
sys.exit(1)
client = OpenAI(
api_key=api_key,
base_url="https://api.devupai.com/v1",
timeout=600,
max_retries=0,
)2Ask for Flex on Chat Completions
service_tier is a Chat Completions field, so every item goes to /v1/chat/completions. The fallback request is the same body without service_tier, which selects the standard tier.
def send(client, item, tier):
"""One chat completion for an item. tier is "flex" or "standard" (no service_tier)."""
request = {
"model": MODEL,
"messages": [
{
"role": "user",
"content": "Classify the sentiment of this review as positive, negative or neutral. "
"Reply with one word.\n\nReview: " + item["prompt"],
}
],
}
if tier == "flex":
request["service_tier"] = "flex"
return client.chat.completions.create(**request)3Fall back or wait, by error code
The error code decides the next step. flex_capacity_unavailable means the request did not start within the Flex wait limit, and model_at_capacity on a Flex request means the model is busy right now. In both cases the standard tier can take the request now, so the program resends it at once instead of waiting. A Flex request that returns 429 is not charged (see Service tiers).
rate_limit_exceeded, tpm_limit_exceeded and concurrency_limit_exceeded are your account's own limits. They apply on every tier, so the program waits and retries on the same tier. On the standard tier, model_at_capacity is retried up to three times after the indicated delay. Any other code is recorded with the item, without a retry.
WORKERS = 4 # requests in flight at the same time
MAX_WAIT_PER_RETRY = 60.0 # seconds, cap for one wait
MAX_TOTAL_WAIT = 300.0 # seconds, cap for all waits of one item
STANDARD_CAPACITY_RETRIES = 3 # retries of model_at_capacity on the standard tier
# On a Flex request, these codes mean Flex has no capacity now: resend on the standard tier.
FALLBACK_CODES = {"flex_capacity_unavailable", "model_at_capacity"}
# The account's own limits: wait, then retry on the same tier.
ACCOUNT_LIMIT_CODES = {"rate_limit_exceeded", "tpm_limit_exceeded", "concurrency_limit_exceeded"} except openai.APIStatusError as e:
code = e.code or f"http_{e.status_code}"
request_id = e.response.headers.get("x-request-id")
# Its outcome is not confirmed yet: record it as unknown, with its request id.
if e.status_code == 409 and code == "request_outcome_pending":
return {"status": "unknown", "code": code, "request_id": request_id}
if e.status_code == 429 and tier == "flex" and code in FALLBACK_CODES:
tier = "standard" # resend at once, without service_tier
continue
can_wait = code in ACCOUNT_LIMIT_CODES or (
code == "model_at_capacity" and capacity_retries < STANDARD_CAPACITY_RETRIES
)
if e.status_code == 429 and can_wait:
delay = retry_delay_seconds(e.response.headers)
if waited + delay <= MAX_TOTAL_WAIT:
if code == "model_at_capacity":
capacity_retries += 1
waited += delay
sleep(delay)
continue
return {"status": "error", "code": code, "request_id": request_id}
# A timed-out request may have run: record it as unknown.
except openai.APITimeoutError:
return {"status": "unknown", "code": "timeout", "request_id": None}
except openai.APIConnectionError:
return {"status": "unknown", "code": "connection_error", "request_id": None}A request that timed out, lost its connection or returned request_outcome_pending (HTTP 409) may have run, so the program records it with status "unknown" and does not send it again. For a 409, the record keeps its x-request-id, so you can confirm the outcome (see request_outcome_pending). A rerun skips items recorded as unknown unless you set resend_unknown (resendUnknown in Node.js).
4Read the wait from the headers
retry-after-ms is read first, then Retry-After in seconds. A missing value, or a value of 0 or below, counts as 1 second. Each wait is capped at 60 seconds and all waits of one item at 300 seconds, so a single item cannot hold a worker indefinitely. The function takes only the headers (in Node.js, the error's headers object, with lower-case names), and the functions that wait take sleep and now as parameters, so you can test them without the network.
def retry_delay_seconds(headers):
"""Seconds to wait before a retry: retry-after-ms first, then Retry-After.
A missing value or a value <= 0 counts as 1 second; one wait is capped at MAX_WAIT_PER_RETRY.
"""
ms = _parse_number(headers.get("retry-after-ms"))
seconds = ms / 1000 if ms is not None else _parse_number(headers.get("retry-after"))
if seconds is None or seconds <= 0:
seconds = 1.0
return min(seconds, MAX_WAIT_PER_RETRY)5Record the served tier, keyed by item id
The response's service_tier field reports the tier that processed the request. The program stores it as received and counts a response without it as unreported.
return {
"status": "ok",
"tier": response.service_tier or "unreported",
"answer": response.choices[0].message.content or "",
"seconds": round(now() - started, 1),
}A few workers run the items. Every request in progress counts toward your account's limits, and a Flex request can be in progress for minutes, so a small pool keeps the batch inside those limits. Results are keyed by item id, so a rerun skips items with status "ok", sends items recorded as "error" again, and sends items recorded as "unknown" again only with resend_unknown (resendUnknown in Node.js).
def run_batch(client, items, results, workers=WORKERS, sleep=time.sleep, now=time.monotonic, resend_unknown=False):
"""Process every item without a finished result and store each outcome in results by item id.
Items with status "unknown" are sent again only with resend_unknown=True.
"""
skip = {"ok"} if resend_unknown else {"ok", "unknown"}
pending = [item for item in items if results.get(item["id"], {}).get("status") not in skip]
with ThreadPoolExecutor(max_workers=workers) as pool:
outcomes = pool.map(lambda item: process_item(client, item, sleep, now), pending)
for item, outcome in zip(pending, outcomes):
results[item["id"]] = outcome
return resultsIn production
- Persist results by item id. Save each outcome as soon as it arrives, in a file or your own database, and load the saved results into
resultsat start, so a rerun skips finished items. An item recorded as"unknown"may have run: forrequest_outcome_pending, check itsx-request-idin the usage export in the dashboard, or ask support. Rerun withresend_unknown(resendUnknownin Node.js) only once you know those items did not run. - Run it as a background job. A Flex item can take minutes before it starts, so run the batch outside any web request handler.
- Log what served each item. Log the served tier of every item and the
x-request-idof every error, so a request can be traced later. - Size the pool to your limits. Set
WORKERSfrom your account's limits on Rate limits, and lower it ifconcurrency_limit_exceededappears often. - Other errors. For a retry policy that covers every error code, see Production error handling.
Errors this recipe handles
Each one has the body {"error":{"message","type":"devup_error","code"}}. The first five are HTTP 429; request_outcome_pending is HTTP 409.
| Code | Headers | What the program does |
|---|---|---|
| flex_capacity_unavailable | Retry-After: 60 | The Flex request did not start within the Flex wait limit. Resend it at once without service_tier (standard tier). |
| model_at_capacity | Retry-After: 5, retry-after-ms: 5000 | On a Flex request: resend at once without service_tier. On the standard tier: wait the indicated delay and retry, up to three times, then record the item with this code. |
| rate_limit_exceeded | Retry-After (seconds) | Your account's request rate. Wait the indicated delay (at least 1 second), then retry on the same tier. |
| tpm_limit_exceeded | Retry-After (seconds) | Your account's tokens-per-minute limit. Wait the indicated delay, then retry on the same tier. |
| concurrency_limit_exceeded | Retry-After: 2 | Your account's limit on requests in progress. Wait the indicated delay, then retry on the same tier. If it repeats, lower WORKERS. |
| request_outcome_pending | x-request-id | HTTP 409: the outcome of the request is not confirmed yet. Record the item with status "unknown" and its x-request-id, without sending it again. A rerun sends it again only with resend_unknown (resendUnknown in Node.js). |
A Flex request that did not start within the wait limit returns:
HTTP/1.1 429 Too Many Requests
Retry-After: 60
x-request-id: <request_id>
{"error":{"message":"Flex capacity is currently unavailable. Retry later, or send the request without service_tier to use the standard tier.","type":"devup_error","code":"flex_capacity_unavailable"}}