What does an idle GPU cost the facility?

What does an idle GPU cost the facility?

Anvilfield estimates $265 per installed GPU-year in idle facility electricity for an eight-H100 node modeled at a constant 1.86 kW, PUE 1.30 and an energy-only rate of $0.10/kWh. Adding a selected 10.2 kW reservation at $150 per reserved IT kW-month brings its annual facility allocation to $2,560 per installed GPU, including non-energy infrastructure costs. The published Brookhaven node study supplies the idle-power input; the overhead ratio, tariff and reservation allowance are stated planning inputs. For delivered work, divide the cost of the whole service window, including its idle periods, by accepted GPU-hours or accepted output.

The worked example shows both the continuously idle annual allocation and a mixed-load month. Equal division across eight GPUs assigns each device a share of the host and facility costs; switching off one GPU leaves shared host loads to be measured.

Keep three ledgers: electricity consumed, capacity committed and work accepted. A tenant with a fixed capacity contract owes its commitment while drawing little energy, and an owner retains infrastructure capital and maintenance costs after shutdown. This model covers facility electricity and a non-energy infrastructure allowance; GPU purchase, software, network services and staffing outside that pool enter the broader data center TCO model.

What counts as an idle GPU?

Define the operating state before attaching watts or cost to it. A powered-off node, a booted node and a service holding a model in memory have different energy and readiness obligations. Anvilfield's workload ledger uses the states below so that “idle” remains a reproducible condition.

NVIDIA's live NVSMI reference defines GPU utilization as the fraction of its sample period during which kernels execute. That measures activity, while acceptance logs establish delivered work. Communication waits, retries and rejected responses explain why scheduler allocation, kernel activity and useful output diverge.

NVIDIA DCGM's profiling documentation, updated October 27, 2026, distinguishes SM activity, occupancy, tensor activity and memory traffic; counter multiplexing can return zeroes during collection. A zero sample alone therefore cannot establish an idle interval. NVSMI also lists GPU and memory utilization queries as unsupported with MIG enabled: retain unavailable readings as unavailable and pair supported partition telemetry with scheduler records.

Set acceptance criteria before a run: model/version, accuracy or training target, request mix and latency objective. MLPerf Inference Rules, live repository edition, sets Closed-division quality targets and scenario-specific performance rules: Server has latency constraints, while Offline measures throughput without a latency constraint. Open-division submissions report achieved accuracy and their latency conditions. Declare the local service's acceptance criteria just as explicitly.

Anvilfield workload-state classification, 2026

Where should power be measured?

Use an external energy meter at every server AC inlet as the node-cost boundary. Sum simultaneous A and B feed measurements and every connected supply; redundant feeds carry portions of the actual load. MLPerf Inference v2.0 power-rules document, updated January 5, 2022, places measurement at the wall and requires synchronized system-under-test and director clocks. Those benchmark principles support this operational method.

Collect board telemetry beside inlet energy for diagnosis. CPUs, DIMMs, NICs, fans and conversion losses contribute to the gap between summed GPU-board power and node input. NVIDIA's NVSMI reference separately defines module power covering a GPU, supported NVIDIA CPU and other module components; adding that GPU again duplicates consumption.

Trace separately fed cooling equipment. The Brookhaven experiment supplied its rear-door heat exchanger from a separate PDU connection, outside the node-power result. A CDU pump or rear-door fan belongs inside exactly one declared IT or facilities boundary; the direct-to-chip cooling guide explains the cooling interfaces that require this mapping.

Reconcile upward from node to rack to IT and facility energy. A rack total includes every connected outlet, so isolate unrelated servers before assigning the entire reading to one workload.

What do measured H100 power and manufacturer ratings actually show?

The Brookhaven eight-H100 node study, arXiv:2412.08602v2, dated December 20, 2024, reports 1.86 kW at idle, equivalent to a 233 W share of whole-node power per installed GPU rather than a measured GPU-board idle draw. Table 1 labels the power column as median power. Its 7.92 kW Llama2-13b training value and 8.43 kW combined GPU/CPU stress value describe different loaded states.

The study separately reports an 8.48 kW stress maximum. Its one-minute samples characterize the experiment's reported behavior; protection and transient tests need instruments matched to the electrical event. Preserve “median,” “maximum observed” and “manufacturer maximum” in labels.

NVIDIA's DGX H100/H200 User Guide, updated October 17, 2026, lists 10.2 kW maximum system power and six 3,300 W supplies in a 4+2 arrangement; adding the supply ratings does not measure server electricity use. Their summed 19.8 kW rating is installed PSU capacity. The tested HGX configuration and the DGX product remain distinct configurations.

NVIDIA's live H100 specification sheet lists configurable maximum TDP up to 700 W for SXM and 350 to 400 W for H100 NVL. Preserve the SKU and form factor when comparing those values.

Telemetry also needs its field definition. NVSMI's supported average board-power field averages the last second on Ampere GPUs other than GA100 and on newer devices; its instantaneous field is the last board measurement. AMD SMI CLI documentation headed 27.1.0 provides power and energy queries and a watch interval in seconds. Retain each vendor's metric scope and update behavior in the export.

How do you run a repeatable idle and load test?

Anvilfield's proposed operational protocol, 2026, uses a ten-minute minimum warmup, distinct idle states, a repeated load sweep and a full application replay. The windows and data-quality decisions below are Anvilfield method choices. Repeat the comparison after driver, firmware, cap, partition or model-residency changes. SPECpower_ssj2008 rules document, updated May 18, 2015, uses configured 240-second load and idle intervals for its Java server benchmark; this GPU field protocol uses longer operational windows.

Capture GPU SKU/count, CPU and memory, BIOS, firmware, driver, cap, clocks, partition state, model residency and cooling state. Record meter accuracy and inlet conditions with the test ID in DCIM monitoring; retain the thermal operating conditions that explain changes in fan or cooling demand.

Define offered load before the sweep. Fifty percent means half the declared reference request rate, with the same model and acceptance criteria. It does not establish half the SM activity, TDP or energy. Include startup, checkpointing, retries and the tail in the complete application replay.

Prefer native cumulative energy when reliable; otherwise integrate timestamped power using the meter's documented averaging behavior. Retain units and reset/wrap events, reject negative counter deltas across resets, and mark unsupported fields and missing samples. A one-second polling loop cannot establish a millisecond peak.

Investigate a higher idle baseline by comparing board and inlet readings at the same residency and thermal state. A board increase points toward changed clocks, memory residency or background GPU work; stable boards with higher inlet power direct the check toward CPUs, fans, networking and power-supply losses.

A usable measured interval has a known configuration, synchronized boundaries and complete required traces. Split measured energy from estimates used to repair gaps; repeat an interval missing its acceptance metric or essential trace. Report meter accuracy and repeat spread separately, and extend replay beyond seven days when the workload cycle is longer.

How do you turn idle watts into facility electricity cost?

Convert measured IT energy into a facility-energy allocation using the matched-period overhead ratio, then multiply by the applicable energy-only rate. The Green Grid defines PUE as total data center energy divided by ICT equipment energy. PUE 1.30 therefore assigns overhead equal to 30% of IT energy, or 23% of the resulting facility total.

ISO/IEC 30134-2:2026, edition 2, published January 16, 2026, covers PUE measurement, calculation, reporting and interpretation. ISO's catalogue identifies the 2016 edition and 2018 amendment as withdrawn; the standard's KPI scope does not set a universal GPU idle wattage. The DOE FEMP July 2024 design guide, section 8.9, recommends continuously trending measurements and archiving at least one year to obtain annual energy totals.

Label a billing-month overhead allocation separately from annual reported PUE. Climate, economizer hours, plant staging, redundancy and IT load change the ratio. Sum matching facility and IT energy totals before calculating an aggregate ratio; an unweighted average of short-period PUE readings can distort the allocation.

Use zone overhead where documented metering supports it. The PUE guide, PUE glossary and cooling-systems overview establish these facility boundaries.

EIA reports 2025 average retail electricity prices of 8.62 cents/kWh industrial and 13.41 cents/kWh commercial. Those are blended retail benchmarks; this model independently selects $0.10/kWh energy-only. Adding demand charges to a blended benchmark risks counting the same cost twice.

Average PUE allocates overhead; shutdown savings require the change in whole-facility energy at comparable work, weather and plant state. Fixed plant loads remain after a small IT reduction. Record matched before/after facility and IT energy and accepted output. Apply no PUE multiplier when measured facility energy already supplies the numerator, and assign separately metered cooling only once.

What facility capacity does an idle GPU reserve?

Reserved capacity is the IT kW committed to a node or tenant in the operating ledger. Anvilfield's selected 10.2 kW reservation at $150 per reserved IT kW-month assigns $1,530 per node-month, or $191 per installed GPU-month. The reservation references NVIDIA's DGX maximum as a planning choice while the energy inputs come from the separate research HGX node.

Finance defines the allowance's cost pool, operations establishes deliverable capacity in the required operating state, and the workload owner supplies the commitment. Redundant infrastructure enters the pool once. Apply neither PUE nor another PSU-count or 2N multiplier to an allowance already defined per reserved IT kW. A facility-side commitment needs its own boundary and deliverable-IT-capacity basis; annual average PUE does not establish peak electrical or cooling headroom.

Physical headroom, unused commitments and intentional failover reserves have different release decisions. The stranded-capacity guide and glossary definition trace binding constraints, while tier classification addresses topology and operating-state expectations. Treat the gap between reservation and idle draw as committed headroom. Permanent reassignment needs a released commitment; temporary sharing needs a documented operating policy that preserves the original service obligation.

Reservation allocation: Anvilfield estimates

How do you derive the allowance and account for unused capacity?

Divide the monthly non-energy facility pool by deliverable IT kW in the required operating state, then assign that rate to each commitment. Identify facility capital recovery, maintenance, rent and facility labor inside the pool. The selected pool here excludes GPU hardware, electricity and separately billed utility demand.

Anvilfield estimates a $150/IT kW-month allowance from a selected $150,000 monthly pool and 1,000 deliverable IT kW. At 800 committed kW, workloads receive $120,000 and finance retains $30,000 for unassigned capacity. The node's $1,530 is part of the assigned total.

A full-recovery policy instead divides that same pool by 800 committed kW, producing $187.50/IT kW-month. Finance chooses the policy and labels the denominator. Reassignment transfers allocation; cash savings require an avoidable expense, a smaller contract or a deferred purchase.

Facility pool reconciliation: Anvilfield estimates

How do demand charges differ from reservation costs?

Energy uses kWh, tariff demand uses kW over the contract's averaging interval, and this reservation allowance uses committed IT kW-month. Record the actual demand window, minimum, ratchet and coincident-peak terms. In Anvilfield's selected arithmetic case, a verified 2 kW reduction in billed demand at $15/kW-month saves $30/month; a reduction outside the determining interval saves $0 when billed demand is unchanged.

Choose one accounting route: energy-only rates plus separate billed demand and an allowance excluding demand, or a reconciled all-in facility pool already containing those costs. This worked model uses the first route with $0 separate allocated demand. Allocate a shared demand invoice by a declared coincident-peak contribution rule and retain the residual centrally; adding tenants' noncoincident peaks can overallocate the invoice. Ratchets and minimums can preserve billed demand after the measured peak falls.

Which cost denominator should you use?

State the numerator before quoting any rate: electricity only, facility allocation or fully loaded compute. Identify allocated overhead separately from avoidable cash expenses. Anvilfield's worked model assigns the same $1,988 monthly facility cost as $0.35 per provisioned GPU-hour, $0.69 per scheduled GPU-hour and $0.77 per useful GPU-hour because the denominators are 5,760, 2,880 and 2,592 hours respectively.

Provisioned time means installed physical GPUs multiplied by the reporting window, including outages. Operationally available time subtracts unavailable GPU-hours; report it separately so outages do not disappear inside a capacity label. Scheduled time follows actual scheduler allocation; useful time follows accepted runs or attributed service intervals. Keep rejected work's costs in the numerator.

Choose the rate that answers the decision. Fleet allocation uses installed time; workload economics needs accepted output as well as accepted time. A token rate requires counted tokens, with input and output mix declared, rather than a conversion from scheduled hours.

Cost denominator table: Anvilfield estimates; job and token rows are accounting rules

How do you calculate useful GPU hours?

Total the physical GPU time assigned to scheduled work, then retain time belonging to accepted runs or explicitly attributed accepted service intervals. Count each run's allocated device count and elapsed hours. Necessary communication and checkpoint time remain inside successful distributed runs; kernel-idle samples do not erase that resource use.

With eight installed GPUs over 720 hours, a 50% scheduled fraction and a 90% time-weighted acceptance fraction produce 2,592 useful GPU-hours under Anvilfield's declared workload-accounting method. All eight GPUs run for 360 hours: 324 accepted node-hours produce 2,592 useful GPU-hours, while 36 rejected node-hours produce 288 excluded GPU-hours.

A request-count success fraction cannot substitute for this time-weighted share when requests have different runtimes or overlap in batches. Attribute resources explicitly, or use accepted jobs or tokens for mixed inference. Retain failed attempts' energy and the ready-idle periods in the same service-window numerator.

Shared GPUs need a declared allocation weight. NVIDIA's MIG profile table, updated November 4, 2026, gives the H100 80 GB 1g.10gb profile one-seventh of SMs and one-eighth of memory. Seven such instances running for one hour sum to one allocated physical GPU-hour under an SM-share policy; compute and memory shares are different accounting weights. Neither weight guarantees equivalent throughput across partitions or GPU models.

Assign shared host/base energy and capacity by a declared reserved-resource share, and assign directly attributable work energy separately. Tenant allocations plus unassigned residual must equal the physical node ledger. For shared inference, prorate overlapping allocations so their weights total no more than one physical GPU at any instant; summing concurrent request durations otherwise overcounts device time. Preserve unsupported partition power as unavailable and use accepted output when equivalent GPU time cannot be established.

Useful-work calculation: Anvilfield estimates

Can a slower job look cheaper per useful GPU-hour?

Yes. Accepted GPU-hours measure resources consumed by successful work. A slower accepted job enlarges that denominator and can lower dollars per GPU-hour while raising dollars per completed result. Compare cost per accepted job or token and time-to-result alongside the hourly rate when changing hardware, caps or batching.

Hold model and tokenizer versions, precision, input-length distribution, output-length policy, concurrency and quality target constant. For inference, retain accepted output tokens separately from input tokens, rejected output and retries; record time to first token, time per output token and chosen latency percentiles. For training, compare the same quality target and report time-to-target. Lower wattage over a longer run can consume more energy.

Worked example: one eight-GPU node for 30 days

Anvilfield estimates $1,988 in monthly facility allocation per node and $0.77 per useful GPU-hour for the inputs below. Idle periods account for $87 of the electricity allocation, while the capacity commitment runs throughout the month. Integrated meter energy replaces the selected constant-state powers in an operating ledger.

Worked inputs and results: Anvilfield estimates

Cross-check against a reported complete training run

The Brookhaven study's Table 1 reports 62 kWh over 480 minutes for Llama2-13b. Dividing that total by eight hours gives 7.75 kW average; multiplying the reported 7.92 kW median by eight hours gives 63.36 kWh. Use the reported total energy for that run and the medians as the declared constant-state planning inputs above.

At the selected PUE 1.30 and $0.10/kWh energy-only rate, Anvilfield estimates $8 of facility electricity for the reported 62 kWh run. Preserve the complete run boundary when joining power and workload records.

Method box: formulas and reproducibility

Multiply each state's mean node power, or adopted constant planning power, by its hours and add the energy. Apply the overhead ratio and energy rate, add capacity and separate demand allocation, then divide by accepted GPU time. For interval pricing, total each interval's allocated energy at its corresponding price.

Anvilfield estimates use the eight-H100 node's idle and median training power reported in arXiv:2412.08602v2, checked in November 2026, as flat power inputs for a 30-day schedule. The schedule, acceptance fraction, PUE, electricity rate and reserved-capacity allowance are stated planning inputs. Calculations retain precision through the final division and round displayed dollars and rates.

Which assumptions move the result most?

Scheduled work and capacity allocation dominate this model's useful-hour result. More scheduled work increases electricity use but spreads the fixed reservation across more accepted hours. The table changes one input at a time while retaining the worked-example inputs otherwise.

Keep scheduling and acceptance separate. Filling an idle window with failed work increases energy without adding accepted output. Treat price and overhead changes as allocation sensitivities; measure marginal plant energy before treating shutdown as cash savings. Electricity reductions alone leave the reservation in place.

Sensitivity table: Anvilfield estimates

For a continuously idle node at the same 1.86 kW and PUE 1.30, Anvilfield estimates annual electricity per installed GPU of $132, $265, $397 and $530 at energy-only rates of $0.05, $0.10, $0.15 and $0.20/kWh respectively. This price range changes the energy line; the selected reservation allowance remains fixed.

When should you consolidate, shut down or cap the node?

Compare equal accepted work over the same service period. Consolidation saves energy after accounting for added loaded energy on the receiving node and the emptied node's residual idle or off-state consumption. Release the emptied node's commitment separately to change its capacity allocation.

A power cap needs a complete-run comparison of energy, accepted output and time-to-result. Retain host overhead during added runtime and readiness between jobs. The power-capping and oversubscription guide covers controls and protection engineering; this ledger supplies the cost comparison.

What is the shutdown break-even rule?

First subtract the energy that staying idle would have used during shutdown/restart from measured transition energy. Divide that net transition penalty by the steady power avoided while off. This division requires off-state power below idle power.

The result is fully-off dwell time; add shutdown and model-reload duration to obtain the required total idle opportunity. A nonpositive net penalty gives a zero energy-only dwell threshold, with readiness requirements still governing availability.

Anvilfield's selected calculation uses 1.86 kW idle, 0.10 kW off-state and 0.50 kWh net transition penalty: avoiding 1.76 kW repays the penalty after 17 minutes fully off. The study supplies the reported idle input; off-state power and transition penalty are selected inputs. For a model-resident service, substitute its measured ready-idle power.

Adding a selected eight-minute shutdown/reload window gives about 25 minutes total idle opportunity. Measure actual readiness and transition energy for the installed configuration.

For monetary shutdown decisions, integrate marginal facility energy and interval prices over the transition and off periods. Include minimum billing obligations and lost accepted output where applicable. The IT energy threshold and average PUE alone do not establish those savings.

Who needs the metering and commissioning work?

Meters, workload-to-meter integration and plant-control changes create a defined instrumentation scope. The estimator prices meters, interfaces and trend verification; operations supplies electrical/cooling boundaries and states; the workload owner supplies acceptance logs. Include those deliverables in electrical commissioning and power-density capacity planning.

Estimators pricing public-facility metering and commissioning packages use past awards in PorchRocket Bid Control to see who won comparable public work and for how much. A priced scope should identify feed count, meter points, retained trends and replay obligations so bids cover the same handoff.

Field checklist: can the cost ledger be reproduced?

An absent boundary, tariff basis or accepted-output log changes the decision. Complete this record before comparing nodes, tenants or operating policies. Preserve raw meter exports, script/configuration versions and the finance allocation pool beside the summary so another reader can reproduce it.

Sources, method and citation

Sources checked: the fourteen primary documents listed below, retrieved with curl on November 6, 2026. Publisher catalogue information establishes the ISO/IEC edition and public scope. The operational protocol and allocation policies are Anvilfield methods.

Use the load protocol, reservation allocation, cost denominators, useful-work calculation, worked example and sensitivity table as specific citation targets. Retain the input conditions when quoting a calculated rate.

Update log: November 5, 2026 editorial date; November 6, 2026 source recheck. Corrected MLPerf scope and glossary links; clarified provisioned hours, shared allocations and shutdown conditions. Rechecked the annual, monthly and sensitivity calculations.

How to cite this page: Anvilfield, What does an idle GPU cost the facility?, 2026, https://anvilfield.com/field-guides/datacenter/gpu-idle-power-facility-cost-model/.

Related tools

FAQ

People also ask

• What is data center TCO?
• What is stranded capacity in a data center?
• What is power capping in a data center?
• What is PUE?
• What does DCIM stand for and what does it manage?

Sources checked

• Brookhaven-led research, Empirical Measurements of AI Training Power Demand on a GPU-Accelerated Node, arXiv:2412.08602v2, December 20, 2024 Reopened with curl November 6, 2026, HTTP 200. Table 1: eight GPUs, median idle/training/stress power, 62 kWh and 480 minutes. Setup: separate rear-door cooling supply. Results: 8.48 kW stress maximum.
• NVIDIA System Management Interface reference, live web edition Reopened with curl November 6, 2026, HTTP 200. Board/module boundaries, utilization semantics, MIG query support and one-second average-power support including GA100 exception.
• NVIDIA DCGM Profiling, updated October 27, 2026 Reopened with curl November 6, 2026, HTTP 200. SM, tensor and memory metrics, collection compatibility and multiplexing zeroes.
• NVIDIA DGX H100/H200 User Guide, introduction, updated October 17, 2026 Reopened with curl November 6, 2026, HTTP 200. 10.2 kW maximum input, six 3,300 W PSUs and 4+2 arrangement; distinct product from tested research node.
• NVIDIA H100 Tensor Core GPU product specifications, live edition Reopened with curl November 6, 2026, HTTP 200. SXM up to 700 W configurable; NVL 350 to 400 W configurable.
• SPECpower_ssj2008 Run and Reporting Rules, updated May 18, 2015 Reopened with curl November 6, 2026, HTTP 200. Active-idle distinction and configured 240-second benchmark intervals; separate from Anvilfield protocol.
• MLCommons MLPerf Inference v2.0 Power Measurement Rules, January 5, 2022 Reopened with curl November 6, 2026, HTTP 200. Wall-power boundary and clock synchronization. Document title/date retained despite mutable repository URL.
• MLCommons MLPerf Inference Rules, live repository text Reopened with curl November 6, 2026, HTTP 200. Closed-division quality targets; Server latency constraints versus Offline throughput; Open-division accuracy and latency reporting. Local acceptance criteria remain explicit.
• The Green Grid, Power Usage Effectiveness definition, undated Reopened with curl November 6, 2026, HTTP 200. Total data center energy divided by ICT equipment energy.
• IEC catalogue, ISO/IEC 30134-2:2026, edition 2.0, January 16, 2026 Reopened with curl November 6, 2026, HTTP 200. Public catalogue scope, publication date and edition only. ISO companion catalogue checked November 6, 2026 for withdrawn 2016/2018 records.
• U.S. DOE FEMP, Best Practices Guide for Energy-Efficient Data Center Design, July 2024 Reopened with curl November 6, 2026, HTTP 200. Sections 8.1 and 8.9: annual energy/PUE boundaries and continuous trends archived for at least one year.
• AMD SMI CLI tool usage, served edition 27.1.0 Reopened with curl November 6, 2026, HTTP 200. Power/energy query flags and watch interval in seconds; no inferred release date.
• U.S. EIA, Electricity explained: Prices and factors affecting prices, 2025 annual averages Reopened with curl November 6, 2026, HTTP 200. 2025 industrial 8.62 cents/kWh and commercial 13.41 cents/kWh are blended retail benchmarks.
• NVIDIA MIG User Guide, Supported MIG Profiles, updated November 4, 2026 Reopened with curl November 6, 2026, HTTP 200. H100 80 GB 1g.10gb: one-seventh SMs, one-eighth memory and seven-instance maximum.