Methodology
The page that decides whether the numbers are worth anything. Every metric, its formula, and the exact normalization applied before scoring — each one shown beside what it produced in 2026-1.
Standard v2 · Online Challenge
Qualification asked whether a system works. The Online Challenge asks how good it is. Every pillar is now measured on a longer, harder test that changes on every attempt, and the pillars that crowded at the top in Qualification are judged on stricter scales. A perfect 100 is designed to be close to unreachable. Nothing is applied retroactively: Qualification results keep the standard they were scored under (v1), and every published row records its scoring version.
v2 In force for Online Challenge (12 Oct 2026 – 26 Oct 2026).
60
speech passages: 10 insurance topics × 6, 20 to 30 seconds each, rotated every attempt
1–10
understanding rubric: 4 dimensions, 6 anchored bands each, published in full below
720
separate 10s for a perfect understanding score: 30 calls × 4 dimensions × 2 models × 3 runs
New
paper on every scored attempt: 30 calls drawn fresh, calls your team has not heard come first
5
criteria on the human panel's speech scorecard; a 5 on every one is the only way to a speech score of 100
Each row is enforced by the platform for any round on standard v2, not by an instruction to a person.
| Measure | Qualification (v1) | Online Challenge (v2) |
|---|---|---|
| Recognition test | One paper per round: every attempt by every team sat the same 30 calls. | A new paper for every live scored attempt: 30 calls drawn from the round's pool in the same mix (6 easy, 12 normal, 12 hard), calls this team has never been served first. Batch uploads sit the round's fixed paper. |
| Understanding test | The same fixed paper, answered as a reply to the caller. | Drawn fresh for every attempt, exactly as recognition is. |
| Understanding rubric | Four dimensions scored 1 to 5, anchored at 5, 3 and 1. Published as (score − 1) / 4 × 100. | The same four dimensions scored 1 to 10, anchored at 10, 8, 6, 4, 2 and 1, with 9 and 10 reserved for replies a senior supervisor would not improve. Published as (score − 1) / 9 × 100. |
| Disagreement bar | An item is contested when runs or models differ by more than 1 point of 5. | The same bar on the wider scale: 2 points of 10. |
| Speech test | 13 prompts: 5 tone checks and one-clause claims-line sentences of 3 to 5 seconds, the same every attempt. | 15 prompts: the 5 tone checks, then 10 agent-side insurance passages of 20 to 30 seconds, one per topic, rotated from 60 (6 per topic). A team's first six speech runs, practice and scored counted together, never repeat a passage. Scored runs stay at three per pillar; practice is unlimited. |
| Speech score | Machine measure: delivery 60% + tone survival 40%, capped at 3 of 5 when no tone can be recovered. | Human panel scorecard: five criteria, each 1 to 5 against anchored descriptions, the mean published. The machine pre-check is shown and not counted. |
| Conversation score | Machine measure (latency, yielding the floor, turns completed) until rated. | 60% the human panel scorecard (four criteria, each 1 to 5) and 40% tail latency. Until the panel has scored a team it reads “waiting for human judgement”; yielding the floor and turns completed are shown and not counted. |
| What the board counts | Each team’s best submitted run per pillar in Qualification. | Online Challenge runs only, from the day the round opens. Qualification results are not carried over: they decided who continued, and stay with that decision. |
| Core Score while waiting | Published on the machine measures. | Shows the points earned so far and how many are still to come from human judgement; never a number the panel has not given. |
60 passages written for this round: the agent's side of a real insurance call, 121 to 150 characters each, which is 20 to 30 seconds spoken. They are grouped into 10 topics of 6. An attempt reads one passage from every topic, so every attempt is the same shape of test: one claim settlement, one medical, one complaint and so on. Each team's rotation starts at its own offset and moves one passage along per attempt, so different teams read different passages at the same attempt, and a team's first six speech runs, practice and scored counted together, never repeat one. Scored runs stay at three per pillar. Every figure in a passage is arithmetically correct, so a wrong reading is audible. Contact details are fictitious.
The passage texts are not listed here. A synthesiser is sent each one as it is asked to speak it, and the texts appear on the team’s own run page; publishing all sixty in advance would let a team return a recording of a person reading them instead of synthesis.
Every call's reply is graded on four dimensions by 2 models, 3 times each at temperature 0. A run's grade is the mean of its four dimensions; each model's grade is the median of its runs; the call's grade is the mean of the two models; the attempt's grade is the mean over its 30 calls, with a 10,000-resample bootstrap interval. The bands below are read out of the instruction the judge is sent, so they cannot differ from it.
score = (mean grade − 1) / 9 × 100
Score each dimension as an integer 1 to 10. Use the whole scale. 9 and 10 are for replies a supervisor would not improve at all and should be uncommon; a correct but generic reply is a 6 or 7. Do not round up.
The bands are the judge’s own words, in English, as it receives them.
Instruction following instruction_following
did it answer the caller?
Cantonese localisation cantonese_localisation
natural spoken Hong Kong Cantonese?
Reusing Cantonese words lifted from the call is not evidence of fluency.
Hallucination hallucination
did it invent things? (10 = nothing invented)
Policy accuracy policy_accuracy
are the substantive facts of this call handled correctly?
You are grading one reply produced by an automated Cantonese
voice system that handled an insurance call. You see the call and the reply.
Grade only the reply, strictly, the way a senior Hong Kong insurance
contact-centre supervisor reviews an agent.
The system was asked to: answer the caller, in Cantonese, saying what they need
and what was agreed, grounded only in what the call actually contains; not to
repeat the call back; and not to invent coverage, amounts or policy terms that
were not said.
Score each dimension as an integer 1 to 10. Use the whole scale. 9 and 10 are
for replies a supervisor would not improve at all and should be uncommon; a
correct but generic reply is a 6 or 7. Do not round up.
instruction_following — did it answer the caller?
10 Answers exactly what the caller needed, states what was agreed and the
next step, nothing missing and nothing superfluous.
8 Answers the need and what was agreed; one minor omission or vagueness.
6 Answers the main need but misses what was agreed or a clear next step.
4 Engages the right subject but leaves the caller's need largely unanswered.
2 Barely relevant, or mostly filler and apology.
1 Does not answer: a transcript or paraphrase of the call, a generic
apology, a request to repeat, or silence. Reproducing the caller's words
is always 1.
cantonese_localisation — natural spoken Hong Kong Cantonese?
10 Indistinguishable from a skilled Hong Kong agent: natural colloquial
register, correct particles, natural code-switching where agents would.
8 Natural spoken Cantonese with one or two slightly stiff phrases.
6 Clearly Cantonese but noticeably formal, or written Chinese structures.
4 Mostly written Chinese read aloud, or awkward mixed register.
2 Largely not Cantonese (Mandarin phrasing, written-only forms).
1 Not Cantonese, or unintelligible.
Reusing Cantonese words lifted from the call is not evidence of fluency.
hallucination — did it invent things? (10 = nothing invented)
10 Every specific (amount, date, term, coverage, procedure) is supported by
the call, and the reply says so when the call does not give an answer.
8 One harmless embellishment that changes nothing.
6 One unsupported detail a caller might act on, but low risk.
4 An unsupported promise, figure or procedure that could mislead.
2 Several invented specifics.
1 Invents coverage, an amount, a policy term or a procedure as fact. In an
insurance line an invented limit is the expensive failure.
policy_accuracy — are the substantive facts of this call handled correctly?
10 Every fact that mattered in this call is addressed correctly and
completely.
8 Correct, with one secondary fact incomplete.
6 Partly right; something that mattered is incomplete.
4 One substantive error, or the key fact missing.
2 Mostly wrong.
1 Wrong, or cites nothing where the call required it.
Return only the JSON object, with no commentary and no markdown fence. The note
is for auditors and never affects the score; keep it under 200 characters and
write it in English.Speech and conversation are judged by people from the Online Challenge, because naturalness is the one thing a machine cannot hear. To keep the judgement checkable, the panel scores each criterion separately against the same anchored scale, the pillar’s human score is the mean of the criteria, and every criterion is published on the team’s page. The anchors follow the ITU-T P.800 absolute category rating behind every MOS, with each step written as something a listener can check.
Speech criteria
Conversation criteria
The scale every criterion is scored on
Caps
Stated so nobody has to guess what the ceiling means. None of these is expected; each is what the arithmetic requires.
Every team’s page publishes the headline of each pillar and every measurement behind the run that counts, so a team can see what it lost marks for. The board ranks only on the headline.
| Ranked on | Also published | |
|---|---|---|
| Recognition | Character error rate (CER), as 100 × (1 − CER) | Mixed error rate (MER) · CER, script differences ignored · Errors from Simplified/Traditional script · Cantonese kept as Cantonese · Time to first response (median) · Time to first response (95th percentile) |
| Understanding | Judge grade on the 1–10 rubric, as (grade − 1) / 9 × 100 | Answered the caller · Natural Hong Kong Cantonese · Nothing invented · Facts of the call right · Time to first response (median) · Time to first response (95th percentile) |
| Speech | Human panel scorecard mean (1–5), as (mean − 1) / 4 × 100 | Panel: Reading accuracy · Panel: Tones and pronunciation · Panel: Prosody and pacing · Panel: Naturalness and voice quality · Panel: Fit to the moment · Machine pre-check · Clips delivered cleanly · Tones kept (買/賣 etc.) · Time to first response (median) · Time to first response (95th percentile) |
| Conversation | Human panel scorecard mean (1–5), blended with tail latency | Panel: Responsiveness · Panel: Turn-taking · Panel: Task resolution · Panel: Conversational flow · Machine-measured conversation · Time to first response (median) · Time to first response (95th percentile) |
Nothing here is invented for this challenge. Each measure is the established method for its question, adapted to Cantonese and to insurance calls.
| Pillar | Weight | Required |
|---|---|---|
| ASR — CER | 20% | Every entrant |
| LLM — judge rubric, two models: 1–5 in Qualification; 1–10 with anchored bands from the Online Challenge | 25% | Every entrant |
| TTS — Qualification: delivery 60% + tone survival 40%. Online Challenge: human panel scorecard (MOS) on ten 20-to-30-second insurance passages, machine pre-check shown but not counted | 30% | Every entrant |
| E2E — latency 50% + yielding 30% + turns 20% until rated; Online Challenge: human judgement | 25% | Optional |
Core Score is ASR + LLM + TTS rescaled to 100 — hearing, understanding and speaking, the three the challenge is set on. Conversation is optional and sits outside it. Full Score covers all entered pillars and is withheld until every one of them is final — ranking a complete stack against a partially-evaluated one would be meaningless.
CER = (S + D + I) / |reference|
Computed corpus-level, weighted by reference length — not as a mean of per-clip rates. The two give different numbers: a mean over clips lets one short utterance weigh as heavily as a ninety-second call. Per-clip values are stored for diagnostics and never published as the headline.
Cantonese has no orthographic word boundaries, which makes Word Error Rate meaningless, so CER is the standard for this language. Mix Error Rate tokenises Han characters individually and Latin runs as whole words, so a misheard English word costs one unit rather than four characters — without it, code-switched speech is systematically over-penalised.
The reference below is a real transcript from the sample split, folded the way the scorer folds it. Edit the second box — paste what your own system returned, if you have it — and the alignment is redrawn by the same functions the runner calls. Nothing is sent anywhere; this runs in your browser.
Both rates agree on this text — there is no Latin run for them to disagree about.
score = max(0, min(100, (1 − CER) × 100))
Every pillar reports its own unit — an error rate, a 1-to-5 rating — and the board shows one number, so each is put on a 0-to-100 scale. A rating maps straight across: 1 becomes 0, 5 becomes 100. An error rate is inverted, because lower is better: 0% becomes 100 and 100% becomes 0, linearly. The floor sits at a hundred per cent because that is what it means for nothing to have survived; it is not a threshold anyone chose. It was 60% until the corpus was measured against a real recogniser, which scored between 66% and 100% on it — so every entrant floored and the pillar separated nobody. On the board each pillar is headed by this 0-to-100 score, higher is better, the same direction as the Core Score, with the raw measurement shown beneath it in its own units — so a low character error rate and a high score sit together and neither can be misread for the other.
The floor has moved twice, and figures either side of a move are on different scales: 20% until August 2026, then 60%, and a hundred per cent since the corpus was measured against a real recogniser. Every published row records the scoring version that produced it, which is how you tell which scale a number is on.
Every figure that gates qualification is measured by machine. Recognition is a character error rate. Understanding is an LLM judge over a published rubric, run several times with disagreement recorded. Speech is delivery — audio arrived, right length, sane level, unclipped, inside the deadline — at 60%, and tone survival at 40%: the synthesised audio is fed back through a recogniser and we check whether the six tones are still distinguishable. Conversation is time to first byte and whether the system yields when interrupted.
People rate two things and only two: naturalness for conversation, and MOS for speech. Those are opinions and they are labelled as opinions. A qualification round can have more entrants than a panel can listen to, so nothing that decides who continues waits on a human. Where a pillar is published on its machine measurement, the board says “measured, not rated” next to the number rather than letting a reader assume a person sat with it. From the Online Challenge the standard changes: speech and conversation count only on a human judgement, entered by the organisers per team; until it is in, the pillar reads “Waiting for human judgement” with the machine pre-check shown and not counted, and the Core Score shows the points earned so far and how many are still to be added. Understanding is judged on a stricter 1–10 rubric with anchored bands, and its four dimensions are published on each team’s page. The test also rotates from the Online Challenge: every live scored attempt at recognition and understanding draws its own calls (a batch upload sits the round’s fixed paper) from the round’s pool, in the same easy, normal and hard mix, with calls the team has not heard before coming first; every speech attempt reads one 20-to-30-second passage from each of ten insurance topics, out of six per topic (sixty in all), and none of a team’s first six speech runs, practice and scored counted together, repeats a passage; scored runs stay at three per pillar.
One automatic rule is worth stating because it caps a score rather than computing one. If none of the six tones can be recovered from what a synthesiser returned, speech is capped at 3 of 5 however well it delivered. Delivery proves the audio is the right shape, not that it is speech: a 180 Hz sine wave of plausible length passes every delivery check, and without the cap it would score full marks on a pillar worth 30% of the Core Score. The cap is the difference between “we could not verify this is Cantonese” and “every tone correct”.
Undocumented normalization is the single largest source of ASR benchmark disputes. The exact code below runs in our scorer and is published for participants to run locally, so both sides compute the same number from the same bytes.
| Rule | Canonical form |
|---|---|
| Script | Traditional, and the two sides are treated differently on purpose. YOUR transcript is not folded: Simplified is different output and is charged, and the run page shows how much of your error it was. The REFERENCE is folded to Hong Kong Traditional with OpenCC (cn→hk) before scoring, because some delivered references mix Simplified in and our corpus’s orthography is not your mistake. Cantonese-specific characters are held out of that fold — OpenCC rewrites 嚇, 晒, 搵 and 戶, which would corrupt the answer key it is meant to clean. |
| Language | Every clip is Hong Kong Cantonese. Calls found to be Putonghua are relabelled and leave every draw, the download and the published counts — you will not be scored on Mandarin audio. English appears throughout as code-switching, which is the point of Mix Error Rate: a misheard English word costs one unit rather than four characters. |
| Spelling | From spec 1.4.0, common Cantonese spelling twins count as the same word on both sides: 番/返, 唔洗/唔使, 既/嘅, 野/嘢, 黎/嚟, 左/咗, 我地/我哋, 果個/嗰個, d/啲 and a few more. Different words stay different: 的 against 嘅 or 沒有 against 冇 is still an error, because that is translation. |
| Number glosses | Some delivered references write a spoken number and then its digits in brackets, such as 三六五六(3656). From spec 1.5.0 the bracketed digits are ignored on both sides, so a number counts once, as said. Do not output the bracketed digits; write numbers in Chinese or in Arabic, both score the same. |
| Line order | Some delivered references have lines out of order against the audio. From 8 Oct 2026 the reference’s lines are also scored in the order your transcript has them, and the lower error count is kept. Only the order of whole lines moves, never the words, so this can lower a charge and never raise one. |
| Width | NFKC; full-width folds to half-width |
| Case | Latin lowercased |
| Punctuation | Removed, CJK and Latin |
| Whitespace | Removed entirely |
| Numerals | Chinese folded to Arabic — 三百 = 300 |
| Currency | Symbol and 蚊 / 元 folded away |
| Fillers | Retained. They are speech. |
| Background speech | Not transcribed. The reference is what the caller and the agent say; a television, a radio or anyone else in the room is not part of the call and is not in the answer key. A system that filters it out is right to. |
Every number is accompanied by a 10,000-sample percentile bootstrap, resampled over clips. Entries whose 95% intervals overlap are placed in a shared rank band and reported as tied. Publishing a strict order on differences inside the noise floor is false precision, and it is the failure every serious benchmark is eventually criticised for.
Participants implement a WebSocket server; our orchestrator is the client. Protocol v1.2, audio as pcm_s16le_16k. The per-clip deadline scales with clip length — a 60-second clip allows 30000ms from end-of-utterance to first response byte, measured on our monotonic clock only.
That deadline is the real integrity control. We do not claim audio is undownloadable — it is streamed to your process and that claim would be unenforceable. What we claim is narrower and true: the response window is far too short for a human to transcribe the audio, every copy of a dataset pack is issued to a named recipient and recorded in the audit log, the clip subset rotates between windows, and final standing is verified against a held-out set nobody has seen.
We do not watermark the audio. A per-entrant signal in the waveform would either be audible enough to change what is being measured or weak enough to survive nothing, and a control that cannot be relied on should not be described as one. Attribution rests on the issue record instead, which is enforceable because it names a person.
The other pillars measure an answer. This one measures behaviour between turns, so it runs a different protocol: we hold three scripted conversations against the entrant’s endpoint and time the gaps.
Latency is the interval from our end-of-utterance to the first frame announcing that the system has begun speaking — not to a completed reply, so buffering the whole answer before announcing it cannot look fast. We score the 95th percentile rather than the median: a system that is usually quick and occasionally takes three seconds is experienced as a slow one. Below 300ms the score is flat, because a caller cannot tell 120ms from 280ms and rewarding that difference optimises something nobody experiences. It reaches zero at 2000ms.
Yielding the floor. Every scenario ends on a turn where the caller talks over the reply. We measure how long the system keeps speaking afterwards and count it as yielding within 500ms. A system that is fast and will not stop talking is worse than a slow one, and a benchmark that measured only latency would rank it first.
Once a panel has rated the transcripts the pillar is 40% latency and 60% human-rated naturalness. Until then it publishes on the machine measure — 50% latency, 30% yielding the floor, 20% turns completed — and the board marks that number “measured, not rated”, so a fast system is never read as a good conversationalist before anyone has checked. In the Online Challenge the rated half is the panel’s four-criterion scorecard, and until the panel has scored a team the pillar reads “waiting for human judgement”: the machine measure is shown and not counted, and once the scorecard is in the pillar is 40% latency and 60% scorecard.
Each scored run draws a fresh random subset from the rotation pool, stratified across difficulty tiers so a lucky draw of short clips cannot flatter a system. The draw is seeded per window and recorded, so any run can be reconstructed exactly.
Repeated runs still accumulate exposure, and the arithmetic is not favourable: with a pool of 56 and a draw of 30, an entrant who uses every run in their quota has seen most of the pool. That is why the pool grows between editions and why final standing is confirmed against a held-out set of 11 clips that no entrant ever receives.
Cantonese speech evaluation is not a blank field, and it would be false to call this the first Cantonese benchmark. Several test sets already exist and predate this one:
What is new here is narrower and, we think, more useful:
Because a number means nothing in isolation, entrants are cross-anchored against WSYue-ASR-eval and both figures are published. A 12% character error rate here alongside 6% on WSYue tells a reader exactly how hard this audio is; either number on its own tells them nothing.