Methodology

What HashBench publishes, how each number is built, and what it does not claim. Every index series on this site is experimental.

What we publish

Three kinds of number, which answer different questions and never share a label:

  • Reference price: what one model release costs across the providers that serve it, in USD per 1M tokens, per direction (input, output, cache read).
  • Basket cost: what a disclosed basket of releases costs today, in USD per 1M tokens.
  • Index level: how prices moved within successive fixed baskets, in points (base 100). A basket change never moves the level by itself.

The scope is the OpenRouter channel: list prices quoted there for standard synchronous text inference, weighted by the tokens OpenRouter reports. It is not the whole global market, and prices are quoted, not transacted.

Reference prices

Five platforms are collected every day (OpenRouter, Artificial Analysis, Hugging Face, Vercel AI Gateway, Portkey). Every row is resolved to one canonical release, creator/model@YYYY-MM-DD, or left unmapped with a reason. Nothing is mapped by guessing.

For every eligible release (uptime and essential-set survivors; the eligible universe currently runs 75-80 releases, ranked and capped at 200), each direction’s reference price is the weighted median of its providers’ base-endpoint list prices. Weights are the tokens each provider served, capped at 35% per provider, with the excess spread over the others. OpenRouter reports a provider’s tokens for the UTC day so far; read at collection, about 15:35 UTC, that covers about 65% of the day. From 10 November 2026 (method 1.4.0) a weight sums that figure over seven days: the day’s own plus the last run’s figure for each of the six days before, on the days the provider was eligible. Before that, each day used its own figure alone. Prices are always that day’s quotes. Providers below 80% trailing-day uptime, OpenRouter’s own threshold for a provider that is down, are excluded, and so are providers whose uptime is not reported (from 1 November 2026, method 1.3.0; before that the floor was 60% and an unknown uptime passed). Published runs are never rewritten; a correction is a new run.

Quantization. A release is one price across its providers, whatever precision each serves. The exception is a provider that lists the same release at more than one quantization (fp4 and fp8, say): from 1 October 2026 (method 1.2.0) that provider counts once, at the quantization whose prices sit closest to the release’s other providers. The choice is made at the first run of each month and held through it; it changes mid-month only if the chosen quantization stops being listed or eligible, and every choice is recorded. Token volume can’t decide it, because OpenRouter reports a provider’s tokens without splitting them by quantization. In the data collected so far this affects one or two provider listings a day out of about a thousand.

A weighted median can jump when routing shifts which provider sits at the 50% mark, even if no tariff changed. Seven-day weights make that rarer but do not remove it. The index therefore does not use the median directly; it prices each release at a fixed provider mix (below). Discovery and the Overview label every move with its cause: tariff (the quoted price that set the median changed or was withdrawn), mixed (a new price appeared while the old one is still on offer) or routing (no price changed).

Index points and dollar costs

Suppose a basket costs $6 per 1M tokens and, at the monthly review, an old release is replaced by a cheaper one so the new basket costs $4. Both facts are true: buying the basket now costs less, and no price the old basket held has moved. The basket cost shows $6 → $4. The index level shows no change at the review, because it links the old basket’s level to the new basket. The reconstitution bridge on each index page publishes all four numbers, so replacement is visible rather than hidden or mistaken for a price cut.

Index family, methodology v0.3

Segments

  • ALL, the broad market: every eligible release.
  • HI, MD, LO: capability tiers. Releases are ranked by their Artificial Analysis Intelligence Index, and a release’s percentile counts the tokens consumed by lower-scored releases (plus half its own). High is roughly the top 20% of token usage, Middle the next 45%, Lower the bottom 35%, with buffers so incumbents do not flap. Reviewed quarterly.
  • OPEN, PROP: open-weights and proprietary releases, per Artificial Analysis’s open-weights flag.

Basket

Each segment holds the smallest set of releases that covers 95% of its token volume. Token weights come from observed volume, adjusted so that no release exceeds 20% (25% in a tier) of the basket’s base-period dollar cost and no creator exceeds 30% (40%). From the 1 December 2026 rebuild, no serving company exceeds 35% (50% in a tier) either; where that cannot hold the basket is flagged rather than forced. Each release is priced at its providers’ quotes in fixed shares frozen at the review, so a routing change cannot move a release rate.

R(release, t) = Σ provider_share × provider_quote(t) C(direction, t) = Σ token_weight × R(release, t) C(blend α, t) = α · C(input, t) + (1 − α) · C(output, t) Level(t) = Level(link) × C(t) / C(link)

Blends are formed from the two dollar baskets before the ratio. Averaging two index levels instead would misstate the move: if input rises 10% and output is flat, a 50/50 average of points says +5%, but the 50/50 basket rises only by input’s share of its dollar cost.

Measures: IN, OUT, and blends MIX80, MIX50, MIX25 (input share of total tokens: 80%, 50%, 25%). These are standard scenarios, not claims about any workload.

Reconstitution: why monthly, and the integrity exception

Core reconstitutes monthly and reviews tiers quarterly on purpose, not by omission. Its job is a stable, comparable benchmark of what capability costs — a series you can read index-point to index-point across months without every reading also carrying a basket change. A slower cadence is what makes that comparison mean something: reconstitute weekly and every week's move is partly "the market changed" and partly "the basket changed," which is fine for a series meant to be rebalanced against (settlement grade, below) and unhelpful for one meant to be read as a single trend. This is also why core carries no incumbency buffer: monthly review already gives a member a full month before falling out, so there is no fast-cadence flapping to dampen.

Since engine 0.3.1, core shares one rule with settlement grade: a member with no fresh quote from any of its providers for more than two days is excluded from that day's cost and coverage and its weight redistributed pro-rata among the rest, rather than carried indefinitely — disclosed via the same fixing fields, stale_symbol and max_carry_days. This is not a cadence or tradability decision; a member whose every provider has gone dark is a data-integrity fact regardless of which family is reading it, and before 0.3.1 that exact case could hold the primary series, AIX.ALL.MIX50, in WITHHELD indefinitely. See docs/index/ASSESSMENT.md section 16. From 1 October 2026 (engine 0.4.0), an excluded member no longer decides the status either: it is not carried, so CARRIED or WITHHELD depends only on the quotes actually carried into that day's cost. From the same date a provider's quote is matched by the provider's name, so a quantization label that changes between reviews no longer leaves its share unmatched. Fixings before that date are unchanged. See docs/index/ASSESSMENT.md section 17.

From 10 November 2026 (engine 0.5.1) the same idea applies to one provider within a member. A provider with no quote for more than seven days, the longest a gated provider's posted rate is held, drops out of that day's rate and coverage, and the member's remaining providers stand in for it by their base-period dollar shares. Its age is still disclosed as the stale member. Before that date, one provider that left a release mid-month was carried until the next rebuild: from 8 November 2026, Decart's departure from DeepSeek V4.1 Flash held 20 series in WITHHELD at 99.97% coverage. See docs/index/ASSESSMENT.md section 25.

Settlement grade, methodology v0.4

The same caps and pricing mechanics as v0.3 — published under their own tickers, AIX.SG.<segment>.<measure>, alongside the core family rather than in place of it. Core answers "what does the market look like"; settlement grade answers the same question at a cadence, buffer and tier definition built to be referenced by a derivative — a future, a swap, a term structure — rather than just observed. Three differences from v0.3 (the integrity exception above is now shared by both, not one of them):

  • Weekly reconstitution, not monthly. This market's constituents turn over far faster than a calendar month assumes — 4 of the top 20 releases by token volume changed in 16 days in September 2026 — so a monthly basket is stale for weeks at a time, which core accepts as the cost of a stable benchmark (see above) but a hedge cannot.
  • An incumbency buffer on selection. A new entrant must still clear the standard 95%-coverage bar; a member already selected the prior week is retained as long as it still ranks within 25% of the member cap, even if it would no longer clear the bar alone. Without this, the faster cadence would turn into marginal-member flapping — the same flapping monthly review avoids for core simply by being slow.
  • PREM / STD / ECO tiers, ranked by price, not HI/MD/LO by capability. A tier is only useful to a hedger if its constituents are close price substitutes, and measured directly (coefficient of variation, max/min ratio of blended price within a tier), the core family's capability tiers aren't especially close ones — the Lower tier's within-tier price spread has run as high as 100x. Ranking the same releases by blended reference price instead cuts that to roughly 3–24x. Reclassified every reconstitution (weekly), since price moves at that speed and capability doesn't.

Full reasoning, the dispersion numbers behind the price-tier choice, and the daily check that keeps tracking them (aix.index.review.check_tier_dispersion): docs/index/v4/AIX_Settlement_Grade_Methodology.md and docs/index/ASSESSMENT.md sections 13–14 and 16.

Equal-model tiers

HIE, MDE, LOE are the capability tiers ranked a second way. The scores, the tier thresholds and buffers, the basket rules and the pricing are exactly those of HI, MD and LO; only the percentile differs. Here each scored release counts once, however many tokens it serves, so a release’s tier says where it stands among models rather than among tokens used. A heavily used model pulls the usage percentile of everything scored above it upward; it moves the equal-model percentile by one step.

They are published as a sensitivity check: where the two rankings put a release in different tiers, the two sets of indices show how much the tier definition, rather than prices, drives a tier index. The measures are the same five.

These replace the methodology v0.2 series (FRT, MID, BDG, HEADLINE), retired on 21 September 2026. Put on v0.3’s basket and pricing, v0.2’s tiers are exactly these: the top tier tracks High within 0.02 points, while the middle and lower tiers differ from Middle and Lower by up to 2.9 and 6.3 points in the first week, because the two rankings put different releases there. The rest of the gap between the published v0.2 and v0.3 series came from v0.2’s construction (top ten by dollar volume, band weights, published medians, a divisor), which v0.3 was written to replace. The count-based ranking was v0.2’s one distinct idea, and it lives on here. Index runs before engine 0.2.0 keep the v0.2 series as they were published.

Market unit value

AIX.UV is the day’s quoted spend divided by its tokens: every provider’s quote for every published release, times the tokens that provider served, summed and divided by the tokens. The tokens are that day’s only, the UTC day so far at collection, not the seven-day weights reference prices use. This is a token-weighted average quoted price (each token counts once, whatever it cost), the construction used by Silicon Data’s LLM Token Expenditure Index. It answers “what did a token cost on average today at list prices, given what people used and where”, and it moves when usage shifts between cheap and expensive models, or between cheap and expensive providers, even if no price changed. The overview sets its change beside the broad index’s change. The gap is a comparison, not a measurement of usage mix alone: the two series also differ in universe, caps and fixed provider shares.

UV(direction, t) = Σ tokens(release, provider, t) × quote(release, provider, t) ÷ Σ tokens(release, provider, t)

Each provider is priced at its own quote rather than at the release’s reference price, because the median is the typical quote, not the average one paid: on 20 September 2026, pricing every token at its release’s median overstated the unit value by about 5%, because, weighted by tokens, providers quote below their release’s reference price on average.

Statuses

✓Fresh
Every component was observed on the fixing day, or its provider was seen but kept out by the eligibility gate, the uptime floor or the essential set (its posted rate is held, up to seven days). From 10 November 2026 a provider with no quote for longer drops out of its member instead of being carried. A member excluded by the integrity exception does not count against freshness; it is disclosed as the stale member.
◐Carried
At least 95% of the basket’s base-period dollar exposure was observed; the rest was carried at most one day. Exposure counts each provider at its price on the link date; before 1 November 2026 a provider’s share was counted by its tokens instead.
✕Withheld
Less than 95% observed, or a quote carried longer than a day. The level is still shown, as an indicative value, because learning how often this happens is part of the experiment.

A fixing also counts review triggers: component quotes that moved more than 20% since last seen. They are not corrected or clipped; they flag days to check.

Caveats you should know

  • Short history. The index starts on 13 September 2026. The runs of 12 September priced some providers’ tier endpoints (fast, flex) as base prices: four releases (Gemini 3.7 Flash, GPT-5.4 Nano, GPT-5.6 Terra, Grok 4.6) show a published price exactly double or half the next day’s. The index excludes them; the reference-price charts shade them and measure changes from 13 September.
  • A method change inside the history. Reference prices up to 17 September weight providers by requests in the last 30 minutes (method 1.0.0); from 20 September by the day’s tokens served (1.1.0); from 10 November by seven days of tokens served (1.4.0). A move across either date can come from the change of weights alone.
  • Corrections from 1 November 2026 (engine 0.5.0). Freshness is measured in base-period dollar exposure, as the methodology states, where it had counted each provider’s token share; the basket cost on a day a member is excluded is the cost of a million tokens of what the basket still holds; and a capability tier is never assigned on the highest of several unresolved Artificial Analysis variants. Earlier fixings are not restated.
  • A departed provider from 10 November 2026 (engine 0.5.1). A provider with no quote for more than seven days drops out of its member until it quotes again, instead of being carried and withholding every series that holds the release. The fixings of 8 and 9 November stay WITHHELD as published.
  • Tiers hold between reviews. A release keeps the capability tier it was placed in at the last review, on that day’s AA score. Artificial Analysis revises its scores, so the AA index shown beside a tier can differ slightly from the score that placed it.
  • Backcast weights. Token volumes are recorded from 20 September 2026. Weights from that day are applied to the base period, which is look-ahead; the baskets that rely on it carry a backcast flag.
  • Total, not directional, volume. OpenRouter reports a provider’s tokens without splitting input from output, so both directions start from one set of token volumes. The dollar caps apply to each direction separately, so the final input and output token weights can differ slightly.
  • Time of day. Some quotes change with the hour of collection: one provider’s price for the same model was exactly double in a 02:00 UTC run what it was in 21:00 UTC runs. Since 30 September 2026 collection runs in one fixed window and finishes about 15:45 UTC; earlier fixing days mix collection hours.
  • Quoted, not paid. List prices exclude caching, batch, negotiated discounts and credits.
  • Not a benchmark for settlement. These series have not passed the launch tests, governance or independent review a financial benchmark requires.

Schedule and data access

Collection runs daily; the index is recomputed from all published reference-price runs after each one. Every number on this site is available as JSON: see the API reference. Index runs are stored whole and never rewritten, like reference-price runs.

Revision policy. No retroactive restatement, ever: a published run — reference price or index — is never edited or deleted. A correction is a new run, dated, standing alongside the one it corrects. Nothing on this site should be read as "the number as of today" superseding "the number as of the day it was fixed" — both remain in the record.

Contributor independence. None of the five reference-price sources (OpenRouter, Artificial Analysis, Hugging Face, Vercel AI Gateway, Portkey) is affiliated with this site or its administrator.

Full governance status, including what's launch-gated rather than missing: docs/index/GOVERNANCE.md.