AWS benchmarks G7 Blackwell GPUs at 60.8% more throughput than G6

AWS published benchmark results on September 8, 2026 that put its new G7 GPU instances ahead of two earlier generations on small Mixture-of-Experts inference. Serving Qwen3-Coder-30B-A3B-Instruct-FP8 on Amazon SageMaker AI, an ml.g7.12xlarge endpoint produced 391.3 output tokens per second. That’s roughly 60.8 percent more than ml.g6.12xlarge and 13.0 percent more than ml.g5.12xlarge, according to the AWS machine learning blog.

The margin matters because of what G7 needed to get there. AWS reports the G5 and G6 configurations each ran four GPUs with 96 GB of aggregate GPU memory, while G7 ran two GPUs with 64 GB. So the winning box had half the accelerators and two thirds of the memory.

Metric	g5.12xl	g6.12xl	g7.12xl
Output token throughput	346.3 tok/s	243.4 tok/s	391.3 tok/s
Request throughput	2.69 req/s	1.89 req/s	3.04 req/s
Average request latency	1,475.2 ms	2,109.7 ms	1,315.8 ms
P99 request latency	1,881.1 ms	3,315.7 ms	1,501.1 ms

The workload was small and AWS says so plainly. It sent 100 requests at a concurrency of four through a DJL Large Model Inference 28.0 container in us-east-2, and collected metrics with NVIDIA AIPerf. Latency tracked throughput: G7 cut average request latency by about 37.6 percent against G6 and 10.8 percent against G5, and P99 latency by 54.7 and 20.2 percent.

Throughput isn’t the whole picture either. On a separate streaming run against the G7 endpoint only, AWS measured average time to first token of 118.9 ms and average inter-token latency of 8.9 ms. Those are the numbers that decide whether a developer copilot feels responsive, which is the use case AWS picked the Qwen model for. The model card lists 30.5B total parameters with 3.3B activated across 128 experts.

Only G7 instances have native FP4 Tensor Core support. Other generations such as G5 and G6 run NVFP4 weights without hardware acceleration, which puts G7 at a structural advantage for MoE deployment choice.

AWS Machine Learning Blog

That hardware gap is what explains the second half of the post. NVFP4 is the 4-bit floating point format NVIDIA introduced with Blackwell, and NVIDIA’s own write-up puts it at roughly 3.5 times less memory than FP16 and 1.8 times less than FP8. NVIDIA also reports 1 percent or less accuracy degradation on key language modelling tasks for DeepSeek-R1-0528 when it moved that model from FP8 to NVFP4.

For the second test AWS handed NVIDIA Nemotron-3-Nano-30B-A3B-NVFP4 to SageMaker’s inference recommendation service and let it pick configurations across G6, G6e and G7 using vLLM, the server Databricks benchmarked its kernel generator against. On a 512-input, 256-output chat profile, ml.g7.48xlarge returned the highest measured throughput at 2,397 output tokens per second. On cost, ml.g7.2xlarge came lowest at an estimated $0.90 per million output tokens, rising to $2.29 on a 3,500-input RAG profile.

The result is a wider gap on price than on speed. AWS puts G7 at roughly 1.4 times lower cost per token than G6e on both profiles, and about 4.9 times lower than G6 for chat and 5.6 times for RAG. The cost figures are derived, not quoted: AWS calculated them from measured throughput and instance price, with throughput as the stated optimisation target.

Even so, the obvious caveat is that this is a vendor measuring its own hardware with its own tooling. AWS states the results are specific to the models, configurations, workloads and Regions tested, and that G7 is generally available in US East (Ohio) and US West (Oregon) only. That regional limit is a real constraint for anyone who isn’t already deployed there, and it’s a familiar one for AWS inference customers, who got cross-Region routing on Bedrock for exactly this reason.

Two things are worth watching. The first is whether G7 capacity reaches more Regions, because a 60.8 percent throughput gain you can’t schedule isn’t worth much. The second is independent reproduction, and AWS has made that cheap by publishing both notebooks in its sagemaker-genai-hosting-examples repository.