Lower power, higher energy: the quantization paradox
Energy and power are easy to confuse. Power is the rate at which the GPU uses energy, measured in watts. Energy is the total cost over time, measured in joules. A quantized model can draw less power but run for longer, so the final energy bill can move in the opposite direction. An always-on GPU also consumes power while it is idle, which raises another choice: should that idle floor be charged to the request? The question was: how do engine choice, quantization, batching, and the idle-power floor change the energy cost of each generated token?
Why this matters
Tokens per second measures speed, not operating cost. Two configurations can rank one way by throughput and another by energy per token. This matters for Green AI, capacity planning, and deciding whether a memory-saving format is also an energy optimization.
Main finding
NF4 drew much less board power than FP16, 64 W versus 116 W, but ran slowly enough to consume 1.21x more gross energy per token. After subtracting the measured 38 W idle floor, NF4 consumed 1.48x less incremental energy. Both answers are correct; they represent different operating assumptions.
Gross energy charges the entire board to the work. Net energy subtracts the measured model-resident idle floor over the same interval.
What we did
I measured Qwen2.5-1.5B inference on a single RTX 3060 12 GB GPU. At batch 1, I compared HuggingFace FP16, HuggingFace NF4, and vLLM FP16. I then swept vLLM batch sizes from 1 to 128.
The primary instrument was NVIDIA’s monotonic total-energy counter, read at the same synchronized boundaries used for timing. A second instrument sampled board power every 20 ms and integrated it as a cross-check. The GPU’s power reading only refreshed about every 503 ms, so direct integration was unreliable for short windows. On the real 5 to 16 second runs, the two methods agreed within 0.5% on 20 of 22 rows.
I also measured the model-resident idle floor after allowing boosted clocks to settle. It remained between 37.7 and 39.3 W across the run groups.
Detailed results
| configuration | throughput | board power | gross energy/token | net energy/token |
|---|---|---|---|---|
| HF FP16, batch 1 | 59.5 tok/s | 116.2 W | 1,960 mJ | 1,337 mJ |
| HF NF4, batch 1 | 26.8 tok/s | 63.6 W | 2,374 mJ | 905 mJ |
| vLLM FP16, batch 1 | 96.0 tok/s | 149.0 W | 1,552 mJ | 1,149 mJ |
| vLLM FP16, batch 128 | 4,136.6 tok/s | 162.1 W | 39 mJ | 30 mJ |
NF4 moved fewer weight bytes, so the board was cooler. But it generated tokens 2.2x more slowly and paid the idle floor for 2.2x longer. If a GPU can sleep as soon as a job finishes, gross energy is the relevant bill and FP16 wins. If the GPU must stay powered between requests, the idle floor is paid either way and NF4’s lower incremental power can win.
Batching moved both throughput and energy efficiency much more than the choice between FP16 and NF4.
The largest result was not the quantization comparison. vLLM fell from 1,552 mJ per generated token at batch 1 to 39 mJ at batch 128, a 40x improvement. Board power stayed near 150 W while many sequences shared each weight read.
What we learned
“Does quantization save energy?” is incomplete without an operating model. A sleepable or continuously busy GPU favors the faster FP16 path in this test. An always-on machine with idle gaps can favor NF4 after the unavoidable floor is separated from incremental work.
Batching is the clearer energy lever. The card consumes almost the same power while advancing one sequence or many, so increasing useful work per decode step spreads nearly fixed joules across more tokens.
Limitations
NVML reports whole-board energy, not separate DRAM and compute components. The study used one GPU, one model, and one group of software versions. CPU and whole-server energy were outside scope. The low-traffic examples are derived from measured medians rather than a live arrival-process experiment.
Future work
Future measurements should include wall-plug server energy, automatic GPU sleep states, larger models, and real request traces. Combining these measurements with serving-load experiments would show how energy changes across an entire day rather than inside isolated generation windows.
Technical evidence
Read the complete report, code, raw measurements, and reproduction instructions on GitHub.