Inference at the Margin: When Latency Becomes a Line Item

The hidden cost of inference

When teams budget for LLM deployment, they think about GPU costs. What they miss is the margin: the delta between when a request arrives and when the first token is produced.

Latency is not just a UX concern. It’s a line item.

Three deployment topologies

Batch processing

Batch is the cheapest topology. You accumulate requests, process them in bulk, and return results. Cost per token is minimal. Latency is measured in minutes or hours.

Best for: document processing, data enrichment, offline analysis.

Streaming

Streaming balances cost and latency. Tokens arrive as they’re generated. The user sees output within milliseconds, even if the full response takes seconds.

Best for: chat interfaces, real-time summarization, interactive coding assistants.

Edge deployment

Edge is the most expensive per-token, but eliminates network latency entirely. The model runs on-device or in a nearby edge node.

Best for: privacy-sensitive workloads, ultra-low-latency requirements, offline-capable applications.

The cost model

Total Cost = GPU Hours + Network Egress + Storage + Latency Premium

Where:
  GPU Hours      = tokens_generated / throughput_per_gpu_hour × price_per_hour
  Network        = tokens × bytes_per_token × price_per_gb
  Storage        = model_size × replicas × price_per_gb_month
  Latency Premium = cost_of_slow_response × p99_latency_seconds

The latency premium is the interesting part. For a customer service bot, each additional second of latency costs approximately $0.50 in abandoned conversations. For a trading system, it’s measured in basis points.

The margin matters

Understanding your deployment topology means understanding where the margin lives. In batch, the margin is throughput. In streaming, it’s time-to-first-token. At the edge, it’s the network hop.

Optimize for the margin that matters to your use case. Everything else is premature optimization.