Methodology
The page that decides whether the numbers are worth anything. Every metric, its formula, and the exact normalization applied before scoring — each one shown beside what it produced in 2026-1.
| Pillar | Weight | Required |
|---|---|---|
| ASR — CER | 20% | Every entrant |
| LLM — judge rubric, two models | 25% | Every entrant |
| TTS — delivery 60% + tone survival 40%, until MOS | 30% | Every entrant |
| E2E — latency 50% + yielding 30% + turns 20%, until rated | 25% | Optional |
Core Score is ASR + LLM + TTS rescaled to 100 — hearing, understanding and speaking, the three the challenge is set on. Conversation is optional and sits outside it. Full Score covers all entered pillars and is withheld until every one of them is final — ranking a complete stack against a partially-evaluated one would be meaningless.
CER = (S + D + I) / |reference|
Computed corpus-level, weighted by reference length — not as a mean of per-clip rates. The two give different numbers: a mean over clips lets one short utterance weigh as heavily as a ninety-second call. Per-clip values are stored for diagnostics and never published as the headline.
Cantonese has no orthographic word boundaries, which makes Word Error Rate meaningless, so CER is the standard for this language. Mix Error Rate tokenises Han characters individually and Latin runs as whole words, so a misheard English word costs one unit rather than four characters — without it, code-switched speech is systematically over-penalised.
The reference below is a real transcript from the sample split, folded the way the scorer folds it. Edit the second box — paste what your own system returned, if you have it — and the alignment is redrawn by the same functions the runner calls. Nothing is sent anywhere; this runs in your browser.
Both rates agree on this text — there is no Latin run for them to disagree about.
score = max(0, min(100, (1 − CER) × 100))
Every pillar reports its own unit — an error rate, a 1-to-5 rating — and the board shows one number, so each is put on a 0-to-100 scale. A rating maps straight across: 1 becomes 0, 5 becomes 100. An error rate is inverted, because lower is better: 0% becomes 100 and 100% becomes 0, linearly. The floor sits at a hundred per cent because that is what it means for nothing to have survived; it is not a threshold anyone chose. It was 60% until the corpus was measured against a real recogniser, which scored between 66% and 100% on it — so every entrant floored and the pillar separated nobody.
The floor has moved twice, and figures either side of a move are on different scales: 20% until August 2026, then 60%, and a hundred per cent since the corpus was measured against a real recogniser. Every published row records the scoring version that produced it, which is how you tell which scale a number is on.
Every figure that gates qualification is measured by machine. Recognition is a character error rate. Understanding is an LLM judge over a published rubric, run several times with disagreement recorded. Speech is delivery — audio arrived, right length, sane level, unclipped, inside the deadline — at 60%, and tone survival at 40%: the synthesised audio is fed back through a recogniser and we check whether the six tones are still distinguishable. Conversation is time to first byte and whether the system yields when interrupted.
People rate two things and only two: naturalness for conversation, and MOS for speech. Those are opinions and they are labelled as opinions. A qualification round can have more entrants than a panel can listen to, so nothing that decides who continues waits on a human. Where a pillar is published on its machine measurement, the board says “measured, not rated” next to the number rather than letting a reader assume a person sat with it.
One automatic rule is worth stating because it caps a score rather than computing one. If none of the six tones can be recovered from what a synthesiser returned, speech is capped at 3 of 5 however well it delivered. Delivery proves the audio is the right shape, not that it is speech: a 180 Hz sine wave of plausible length passes every delivery check, and without the cap it would score full marks on a pillar worth 30% of the Core Score. The cap is the difference between “we could not verify this is Cantonese” and “every tone correct”.
Undocumented normalization is the single largest source of ASR benchmark disputes. The exact code below runs in our scorer and is published for participants to run locally, so both sides compute the same number from the same bytes.
| Rule | Canonical form |
|---|---|
| Script | Traditional, and the two sides are treated differently on purpose. YOUR transcript is not folded: Simplified is different output and is charged, and the run page shows how much of your error it was. The REFERENCE is folded to Hong Kong Traditional with OpenCC (cn→hk) before scoring, because some delivered references mix Simplified in and our corpus’s orthography is not your mistake. Cantonese-specific characters are held out of that fold — OpenCC rewrites 嚇, 晒, 搵 and 戶, which would corrupt the answer key it is meant to clean. |
| Language | Every clip is Hong Kong Cantonese. Calls found to be Putonghua are relabelled and leave every draw, the download and the published counts — you will not be scored on Mandarin audio. English appears throughout as code-switching, which is the point of Mix Error Rate: a misheard English word costs one unit rather than four characters. |
| Width | NFKC; full-width folds to half-width |
| Case | Latin lowercased |
| Punctuation | Removed, CJK and Latin |
| Whitespace | Removed entirely |
| Numerals | Chinese folded to Arabic — 三百 = 300 |
| Currency | Symbol and 蚊 / 元 folded away |
| Fillers | Retained. They are speech. |
| Background speech | Not transcribed. The reference is what the caller and the agent say; a television, a radio or anyone else in the room is not part of the call and is not in the answer key. A system that filters it out is right to. |
Every number is accompanied by a 10,000-sample percentile bootstrap, resampled over clips. Entries whose 95% intervals overlap are placed in a shared rank band and reported as tied. Publishing a strict order on differences inside the noise floor is false precision, and it is the failure every serious benchmark is eventually criticised for.
Participants implement a WebSocket server; our orchestrator is the client. Protocol v1.2, audio as pcm_s16le_16k. The per-clip deadline scales with clip length — a 60-second clip allows 30000ms from end-of-utterance to first response byte, measured on our monotonic clock only.
That deadline is the real integrity control. We do not claim audio is undownloadable — it is streamed to your process and that claim would be unenforceable. What we claim is narrower and true: the response window is far too short for a human to transcribe the audio, every copy of a dataset pack is issued to a named recipient and recorded in the audit log, the clip subset rotates between windows, and final standing is verified against a held-out set nobody has seen.
We do not watermark the audio. A per-entrant signal in the waveform would either be audible enough to change what is being measured or weak enough to survive nothing, and a control that cannot be relied on should not be described as one. Attribution rests on the issue record instead, which is enforceable because it names a person.
The other pillars measure an answer. This one measures behaviour between turns, so it runs a different protocol: we hold three scripted conversations against the entrant’s endpoint and time the gaps.
Latency is the interval from our end-of-utterance to the first frame announcing that the system has begun speaking — not to a completed reply, so buffering the whole answer before announcing it cannot look fast. We score the 95th percentile rather than the median: a system that is usually quick and occasionally takes three seconds is experienced as a slow one. Below 300ms the score is flat, because a caller cannot tell 120ms from 280ms and rewarding that difference optimises something nobody experiences. It reaches zero at 2000ms.
Yielding the floor. Every scenario ends on a turn where the caller talks over the reply. We measure how long the system keeps speaking afterwards and count it as yielding within 500ms. A system that is fast and will not stop talking is worse than a slow one, and a benchmark that measured only latency would rank it first.
Once a panel has rated the transcripts the pillar is 40% latency and 60% human-rated naturalness. Until then it publishes on the machine measure — 50% latency, 30% yielding the floor, 20% turns completed — and the board marks that number “measured, not rated”, so a fast system is never read as a good conversationalist before anyone has checked.
Each scored run draws a fresh random subset from the rotation pool, stratified across difficulty tiers so a lucky draw of short clips cannot flatter a system. The draw is seeded per window and recorded, so any run can be reconstructed exactly.
Repeated runs still accumulate exposure, and the arithmetic is not favourable: with a pool of 56 and a draw of 30, an entrant who uses every run in their quota has seen most of the pool. That is why the pool grows between editions and why final standing is confirmed against a held-out set of 11 clips that no entrant ever receives.
Cantonese speech evaluation is not a blank field, and it would be false to call this the first Cantonese benchmark. Several test sets already exist and predate this one:
What is new here is narrower and, we think, more useful:
Because a number means nothing in isolation, entrants are cross-anchored against WSYue-ASR-eval and both figures are published. A 12% character error rate here alongside 6% on WSYue tells a reader exactly how hard this audio is; either number on its own tells them nothing.