The architectural decisions you make in the first two weeks of building a generative media application are the ones that will constrain you — or free you — eighteen months later. Generation latency, cost-per-output, content moderation pipeline position, caching strategy: none of these are easy to re-architect once you have real users. This post is about the patterns we've seen work across different types of generative media apps, and the anti-patterns that look reasonable early on but create compounding problems at scale.
We're writing from the perspective of a decentralized inference provider, which means our frame is specifically around applications that rely on external inference — not self-hosted models. If you're running your own inference cluster, some of this applies; much of it doesn't.
Pattern 1: Treat Generation as an Async Bounded Job, Not a Synchronous HTTP Request
The most common early mistake is building generation into a synchronous request-response cycle at the application layer. A user submits a generation request, your frontend waits on a long-poll or WebSocket, the image comes back, you display it. This works fine at low concurrency and with fast models. It breaks in several ways as you scale.
First, generation latency is variable. At median load, FLUX.1-schnell returns in under two seconds. At p95, under burst demand, the same model might take six to eight seconds. If your frontend is blocking on that response, that six-second tail becomes part of your user's perceived wait time. If you're doing synchronous generation inside a server-side API endpoint, those slow responses tie up server threads.
The better pattern: submit the generation job, return a job ID immediately to the frontend, poll or subscribe to a completion event. This decouples your UI responsiveness from inference latency. The user can see a loading state, a progress indicator, or continue using the app while generation completes in the background. When the result is ready, push it to the client via WebSocket, SSE, or a simple polling endpoint.
// Submit and return immediately
const job = await sogniClient.jobs.create({
model: "flux-schnell",
prompt: userPrompt,
size: "1024x1024"
});
// Return job ID to frontend
return { jobId: job.id, status: "pending" };
// Frontend polls /jobs/:id every 1.5s until status === "completed"Pattern 2: Model Routing at the Request Layer
Not every generation request in your application has the same quality requirements. An avatar preview thumbnail does not need the same fidelity as a hero image export. A real-time filter effect needs minimum latency; a high-res print export can tolerate 15 seconds. Building a single generation pipeline that uses the same model and same parameter set for every request is leaving both quality and cost on the table.
Implement a model routing layer — even a simple one — that selects model, resolution, and step count based on the generation context:
def select_generation_config(context: GenerationContext) -> GenerationConfig:
if context.type == "preview":
return GenerationConfig(
model="flux-schnell",
size="512x512",
steps=4,
credits_estimate=1
)
elif context.type == "standard_export":
return GenerationConfig(
model="sdxl-base",
size="1024x1024",
steps=30,
credits_estimate=4
)
elif context.type == "high_res_export":
return GenerationConfig(
model="flux-dev",
size="2048x2048",
steps=25,
credits_estimate=16
)This pattern also gives you a natural place to insert cost controls — if a user's credit balance is low, you can route to the cheaper preview config and surface an upgrade prompt, rather than rejecting the request outright.
Pattern 3: Output Caching Is Not Optional
A surprising portion of generation requests in production apps are semantically identical or near-identical to previous requests. Consider an e-commerce product visualizer where users select from a finite set of product SKUs and style presets. If 50 users all request "SKU-4721 in the 'outdoor lifestyle' style at 1024×1024," those are functionally identical jobs. Without caching, you're generating — and paying for — the same output 50 times.
Exact-match caching on a hash of (model, prompt, negative_prompt, seed, size, steps, cfg_scale) is the simplest layer and catches the obvious duplicates. More sophisticated applications add a semantic similarity layer: if a new prompt is cosine-similar above a threshold to a previously generated prompt with the same model and seed, serve the cached result. This requires an embedding store and adds latency to the cache-check path, so it's worth profiling whether the savings justify the complexity for your specific traffic pattern.
Cache TTL strategy matters too. Generated images from fixed prompts don't become stale, so a long TTL (days to weeks) is appropriate for deterministic-prompt results. Personalized generations tied to user-provided prompts have lower cache hit rates and you can afford shorter TTLs without losing much efficiency.
Pattern 4: Content Moderation Pipeline Position
Where you place content moderation in your pipeline significantly affects both user experience and safety guarantees. The two extreme positions are pre-moderation (block the prompt before sending to inference) and post-moderation (generate first, then classify the output). Both have tradeoffs.
Pre-moderation with a fast text classifier on the prompt is cheap and non-blocking — it runs in milliseconds and prevents the generation entirely if the prompt violates policy. The limitation: text classifiers have false positives, and blocking a legitimate creative prompt because it contains a flagged keyword creates friction for professional users. It also doesn't catch adversarial prompts that evade text filtering.
Post-moderation on the generated image is more accurate but means you've already consumed the compute credit. For most consumer applications, a combined approach works well: lightweight text pre-filter for obvious violations, image classifier post-filter for everything that passes text screening, with the image filter threshold tuned for your audience and content policy.
We're not saying one approach is universally correct — the right pipeline position depends entirely on your content policy, user base, and tolerance for false positive friction versus false negative risk.
Pattern 5: Credit Budget Enforcement Before the API Call
If your application manages generation credits on behalf of users — either through subscription limits or per-generation charging — enforce budget checks before submitting the job to the inference API, not after. This sounds obvious but is frequently implemented wrong: apps submit the job, get a successful generation, then discover the user's balance was insufficient and either swallow the cost or have an awkward post-hoc conversation with the user.
The correct flow:
- Estimate the credit cost of the requested generation (model + resolution + steps)
- Check user's available balance against that estimate with a small buffer (10–15%) for estimation variance
- Reserve (not deduct) the estimated credits, marking them as pending
- Submit the generation job
- On completion, deduct actual cost and release the reservation delta
- On failure, release the entire reservation
This reservation-and-release pattern prevents double-spend, handles failures gracefully, and never surprises users with a deduction they didn't budget for. It adds one database write to the critical path but eliminates a whole category of billing edge cases.
Pattern 6: Prompt Versioning for Stable Features
If your application uses AI generation as part of a product feature — not just user-free-form generation — you'll eventually face the problem of model updates changing output characteristics. A prompt that reliably produces a certain visual style with model version N may produce noticeably different results with version N+1.
Store your production prompt templates with an explicit model version pinned. When you want to update the model, treat it as a feature flag rollout: test the new model against your prompt library, verify output acceptability, then gradually shift traffic. This is standard practice in any ML-serving infrastructure but is often overlooked in generative media apps where "we're just calling an API" creates a false sense of insulation from model versioning concerns.
The Pattern That Spans All of Them
Every pattern above shares a common underlying principle: treat generation as a first-class, high-value, potentially-expensive async operation — not as a fast, cheap I/O call. The apps that run into trouble are the ones that treat inference like a database read: synchronous, fast, always available, negligible cost. The apps that scale well are the ones that build generation into their architecture with the same care they'd give a payment processing flow — with queuing, error handling, cost tracking, and graceful degradation built in from the start.