Performance benchmarks for inference infrastructure are frequently published by the vendors running the infrastructure, which is an obvious credibility problem. We'll be direct about what this analysis is and isn't: it's an internal measurement of generation latency, throughput, and cost across a set of diffusion and video models running on our own network under controlled and load conditions. We're sharing the methodology and the patterns we found, not a third-party audit. Read it as an engineering reference, not marketing.
The main finding worth stating up front: distributed routing consistently reduces p95 latency at sustained load compared to single-node centralized inference. The median (p50) difference is smaller than you might expect. The tail matters more than the median for user-facing applications, and the tail behavior is where decentralized routing shows the clearest advantage.
Test Setup and Methodology
We ran generation requests across a representative subset of models, varying concurrency from 1 to 50 simultaneous requests. For each model and concurrency level, we collected 200 samples and computed p50, p95, and p99 latency, alongside throughput (jobs/minute) and effective cost-per-generation. Tests were run in February 2025 from a request origin in Singapore; node pool included machines in Southeast Asia, East Asia, and Europe.
The "centralized" baseline was a single A10G instance (24GB VRAM) running each model with no queue competition — i.e., a best-case single-node scenario, not a shared cloud API. This baseline is intentionally favorable to centralized architecture; we wanted to isolate the routing advantage, not the queue-contention effect of shared cloud APIs.
We're not publishing the full raw data tables here — the numbers are specific to our network state at testing time, and they will change as the node pool evolves. What we're publishing is the patterns, which are more durable than point-in-time millisecond counts.
Image Generation: Latency Results by Model
For SDXL-base at 1024×1024, 30 steps, single concurrent request: p50 latency on the distributed network was comparable to the single A10G node — within about 200ms, which is routing overhead. At 20 concurrent requests, the picture changed: the single node's p95 latency climbed to 18–22 seconds (queue depth effect), while the distributed network maintained p95 under 5 seconds by spreading load across multiple nodes. The p99 on the single node at 20 concurrency exceeded 35 seconds — functionally a timeout scenario for most applications.
For FLUX.1-schnell (4-step distilled model), the absolute latency numbers are much lower — p50 around 900ms–1.2s at low concurrency. The concurrency scaling pattern is similar: single-node p95 degrades faster under load than the distributed network's routing can compensate. At 50 concurrent requests, distributed routing maintained p95 under 3.5 seconds for schnell; single-node p95 had climbed to 28+ seconds.
The p95 vs p50 Distinction Matters for App Design
If your application shows a loading spinner and completes generation in under 2 seconds most of the time, users will have an acceptable experience — even if a small percentage of requests take longer. But if your p95 is 20+ seconds under load, you'll have a tail of user experiences that feel broken. Users who hit the p95 will churn; users who hit the p50 won't notice.
This is why distributed routing's primary value is in the tail, not the median. Developers optimizing for p50 latency may not see a meaningful difference between architectures. Developers optimizing for p95 at production concurrency will see a very different picture.
Video Generation: Different Dynamics
Video generation has fundamentally different latency characteristics because the output artifact is much larger and generation steps run across temporal dimension as well as spatial. A 5-second, 512×512 video at standard quality takes 60–120 seconds on a single mid-range GPU. This is not a workload where you optimize for sub-2-second response — users accept longer generation times for video, and the interaction pattern is async by necessity.
The distributed routing advantage for video is less about latency and more about throughput and cost. A network with 50 available nodes can process 50 concurrent video generation jobs simultaneously; a single high-end node processes them sequentially. For a video generation platform with real users, the throughput ceiling on single-node architecture becomes the product bottleneck very quickly.
Cost per video generation is also where the distributed model shows its clearest advantage. Running a dedicated H100 instance 24/7 to serve variable video generation demand means paying for peak capacity even during off-peak hours. Distributed routing scales credit consumption with actual job volume — no idle capacity charges.
Throughput and Concurrency Headroom
One way to characterize network throughput is maximum sustainable jobs-per-minute without p95 exceeding a threshold. We used 5 seconds as our threshold for image generation and 180 seconds for video.
For SDXL image generation, a single A10G node can sustain roughly 8–12 jobs/minute within the 5-second p95 threshold. Beyond that, queue depth causes p95 to spike. The distributed network's ceiling (at our node pool size at testing time) was substantially higher — the practical ceiling shifted from node compute capacity to network scheduling overhead, which is more addressable through software optimization than hardware procurement.
We're not going to publish a specific jobs/minute number for the distributed network because it's a function of current node pool size, which changes. The point is that the ceiling is different in kind: single-node scaling requires adding hardware; distributed routing scales by adding nodes, which happens organically as more providers contribute compute.
Cost Analysis
Cost comparison depends heavily on the benchmark you're comparing against. If you're comparing against reserved-instance cloud pricing for a GPU optimized for ML inference, decentralized inference runs meaningfully cheaper per generation — the structural cost advantage described in our economics posts applies here empirically. If you're comparing against spot instances during off-peak hours, the gap narrows.
One cost factor often overlooked in benchmark comparisons: at what concurrency does your application actually operate? If you're running 2–5 concurrent jobs, the cost-per-generation of a centralized API with reasonable pricing is not dramatically different from a decentralized network. The cost advantage compounds at scale — the developers who see the largest cost benefit are the ones running thousands of generations per day, where the per-generation difference becomes a significant absolute monthly figure.
What These Numbers Don't Tell You
Latency and throughput benchmarks don't capture everything relevant to an infrastructure choice. Output quality consistency across nodes is a real variable — when you're routing to different hardware with different precision characteristics, subtle variation in outputs is possible. We verify outputs via hash-sampling and run model fingerprint checks to ensure nodes are running the declared model, but this is not a solved problem across the industry and deserves honest acknowledgment.
Geographic distribution of the node pool also affects latency in ways our Singapore-origin test can't generalize. A developer in São Paulo or Berlin will see different absolute numbers. The p95 vs p50 patterns should hold; the absolute millisecond figures will vary by origin-to-node round-trip.
Use this analysis as a framework for your own testing, not as a substitute for it. The methodology — test at your actual expected concurrency, measure p95 not just p50, compare total cost at scale not per-request cost — is the transferable part.