Cantonese Voice Benchmark粵語語音基準

Methodology

How we measure.

The page that decides whether the numbers are worth anything. Every metric, its formula, and the exact normalization applied before scoring — each one shown beside what it produced in 2026-1.

Standard v2 · Online Challenge

Harder than Qualification, on purpose.

Qualification asked whether a system works. The Online Challenge asks how good it is. Every pillar is now measured on a longer, harder test that changes on every attempt, and the pillars that crowded at the top in Qualification are judged on stricter scales. A perfect 100 is designed to be close to unreachable. Nothing is applied retroactively: Qualification results keep the standard they were scored under (v1), and every published row records its scoring version.

v2 In force for Online Challenge (12 Oct 2026 – 26 Oct 2026).

60

speech passages: 10 insurance topics × 6, 20 to 30 seconds each, rotated every attempt

1–10

understanding rubric: 4 dimensions, 6 anchored bands each, published in full below

720

separate 10s for a perfect understanding score: 30 calls × 4 dimensions × 2 models × 3 runs

New

paper on every scored attempt: 30 calls drawn fresh, calls your team has not heard come first

5

criteria on the human panel's speech scorecard; a 5 on every one is the only way to a speech score of 100

What changed from Qualification

Each row is enforced by the platform for any round on standard v2, not by an instruction to a person.

MeasureQualification (v1)Online Challenge (v2)
Recognition testOne paper per round: every attempt by every team sat the same 30 calls.A new paper for every live scored attempt: 30 calls drawn from the round's pool in the same mix (6 easy, 12 normal, 12 hard), calls this team has never been served first. Batch uploads sit the round's fixed paper.
Understanding testThe same fixed paper, answered as a reply to the caller.Drawn fresh for every attempt, exactly as recognition is.
Understanding rubricFour dimensions scored 1 to 5, anchored at 5, 3 and 1. Published as (score − 1) / 4 × 100.The same four dimensions scored 1 to 10, anchored at 10, 8, 6, 4, 2 and 1, with 9 and 10 reserved for replies a senior supervisor would not improve. Published as (score − 1) / 9 × 100.
Disagreement barAn item is contested when runs or models differ by more than 1 point of 5.The same bar on the wider scale: 2 points of 10.
Speech test13 prompts: 5 tone checks and one-clause claims-line sentences of 3 to 5 seconds, the same every attempt.15 prompts: the 5 tone checks, then 10 agent-side insurance passages of 20 to 30 seconds, one per topic, rotated from 60 (6 per topic). A team's first six speech runs, practice and scored counted together, never repeat a passage. Scored runs stay at three per pillar; practice is unlimited.
Speech scoreMachine measure: delivery 60% + tone survival 40%, capped at 3 of 5 when no tone can be recovered.Human panel scorecard: five criteria, each 1 to 5 against anchored descriptions, the mean published. The machine pre-check is shown and not counted.
Conversation scoreMachine measure (latency, yielding the floor, turns completed) until rated.60% the human panel scorecard (four criteria, each 1 to 5) and 40% tail latency. Until the panel has scored a team it reads “waiting for human judgement”; yielding the floor and turns completed are shown and not counted.
What the board countsEach team’s best submitted run per pillar in Qualification.Online Challenge runs only, from the day the round opens. Qualification results are not carried over: they decided who continued, and stay with that decision.
Core Score while waitingPublished on the machine measures.Shows the points earned so far and how many are still to come from human judgement; never a number the panel has not given.

The speech test

60 passages written for this round: the agent's side of a real insurance call, 121 to 150 characters each, which is 20 to 30 seconds spoken. They are grouped into 10 topics of 6. An attempt reads one passage from every topic, so every attempt is the same shape of test: one claim settlement, one medical, one complaint and so on. Each team's rotation starts at its own offset and moves one passage along per attempt, so different teams read different passages at the same attempt, and a team's first six speech runs, practice and scored counted together, never repeat one. Scored runs stay at three per pillar. Every figure in a passage is arithmetically correct, so a wrong reading is audible. Contact details are fictitious.

The passage texts are not listed here. A synthesiser is sent each one as it is asked to speak it, and the texts appear on the team’s own run page; publishing all sixty in advance would let a team return a recording of a person reading them instead of synthesis.

TopicP1P2P3P4P5P6
1 Claim settlement123456
2 Policy details and renewal612345
3 Tone-sensitive words561234
4 Medical terms456123
5 Declining a claim345612
6 Digital self-service234561
7 Premium arithmetic123456
8 Incident narrative612345
9 Legal and beneficiaries561234
10 Complaints and service recovery456123
Which attempt reads which passage, for one example team. Each row is one topic; each column one of its passages. The number is the attempt that reads it.
1. Claim settlement
Amounts worked out aloud: deductibles, sub-limits, percentages, currency conversion.
2. Policy details and renewal
Policy numbers mixing letters and digits, dates, sums insured, renewal premiums.
3. Tone-sensitive words
Minimal pairs in one breath: 報/保, 買/賣, 賠/陪, 理財/理賠, 退保/投保.
4. Medical terms
Diagnoses, procedures and English inside Cantonese: VHIS, MRI, X-ray, oncology, cardiology.
5. Declining a claim
Bad news with care: waiting periods, exclusions, non-disclosure, lapsed policies.
6. Digital self-service
App and portal steps, OTPs, email addresses and English interface words.
7. Premium arithmetic
Payment modes, loadings, discounts, policy loans and surrender values, every sum stated.
8. Incident narrative
Times, places and sequences: accidents, robbery, water damage, work injury.
9. Legal and beneficiaries
Beneficiaries, trusts, powers of attorney, assignment and intestacy.
10. Complaints and service recovery
Apology, escalation, case and reference numbers, regulators.

The understanding rubric, in full

Every call's reply is graded on four dimensions by 2 models, 3 times each at temperature 0. A run's grade is the mean of its four dimensions; each model's grade is the median of its runs; the call's grade is the mean of the two models; the attempt's grade is the mean over its 30 calls, with a 10,000-resample bootstrap interval. The bands below are read out of the instruction the judge is sent, so they cannot differ from it.

score = (mean grade − 1) / 9 × 100

Score each dimension as an integer 1 to 10. Use the whole scale. 9 and 10 are for replies a supervisor would not improve at all and should be uncommon; a correct but generic reply is a 6 or 7. Do not round up.

The bands are the judge’s own words, in English, as it receives them.

Instruction following instruction_following

did it answer the caller?

  1. 10Answers exactly what the caller needed, states what was agreed and the next step, nothing missing and nothing superfluous.
  2. 8Answers the need and what was agreed; one minor omission or vagueness.
  3. 6Answers the main need but misses what was agreed or a clear next step.
  4. 4Engages the right subject but leaves the caller's need largely unanswered.
  5. 2Barely relevant, or mostly filler and apology.
  6. 1Does not answer: a transcript or paraphrase of the call, a generic apology, a request to repeat, or silence. Reproducing the caller's words is always 1.

Cantonese localisation cantonese_localisation

natural spoken Hong Kong Cantonese?

  1. 10Indistinguishable from a skilled Hong Kong agent: natural colloquial register, correct particles, natural code-switching where agents would.
  2. 8Natural spoken Cantonese with one or two slightly stiff phrases.
  3. 6Clearly Cantonese but noticeably formal, or written Chinese structures.
  4. 4Mostly written Chinese read aloud, or awkward mixed register.
  5. 2Largely not Cantonese (Mandarin phrasing, written-only forms).
  6. 1Not Cantonese, or unintelligible.

Reusing Cantonese words lifted from the call is not evidence of fluency.

Hallucination hallucination

did it invent things? (10 = nothing invented)

  1. 10Every specific (amount, date, term, coverage, procedure) is supported by the call, and the reply says so when the call does not give an answer.
  2. 8One harmless embellishment that changes nothing.
  3. 6One unsupported detail a caller might act on, but low risk.
  4. 4An unsupported promise, figure or procedure that could mislead.
  5. 2Several invented specifics.
  6. 1Invents coverage, an amount, a policy term or a procedure as fact. In an insurance line an invented limit is the expensive failure.

Policy accuracy policy_accuracy

are the substantive facts of this call handled correctly?

  1. 10Every fact that mattered in this call is addressed correctly and completely.
  2. 8Correct, with one secondary fact incomplete.
  3. 6Partly right; something that mattered is incomplete.
  4. 4One substantive error, or the key fact missing.
  5. 2Mostly wrong.
  6. 1Wrong, or cites nothing where the call required it.
Read the exact instruction the judge receives
You are grading one reply produced by an automated Cantonese
voice system that handled an insurance call. You see the call and the reply.
Grade only the reply, strictly, the way a senior Hong Kong insurance
contact-centre supervisor reviews an agent.

The system was asked to: answer the caller, in Cantonese, saying what they need
and what was agreed, grounded only in what the call actually contains; not to
repeat the call back; and not to invent coverage, amounts or policy terms that
were not said.

Score each dimension as an integer 1 to 10. Use the whole scale. 9 and 10 are
for replies a supervisor would not improve at all and should be uncommon; a
correct but generic reply is a 6 or 7. Do not round up.

instruction_following — did it answer the caller?
  10  Answers exactly what the caller needed, states what was agreed and the
      next step, nothing missing and nothing superfluous.
  8   Answers the need and what was agreed; one minor omission or vagueness.
  6   Answers the main need but misses what was agreed or a clear next step.
  4   Engages the right subject but leaves the caller's need largely unanswered.
  2   Barely relevant, or mostly filler and apology.
  1   Does not answer: a transcript or paraphrase of the call, a generic
      apology, a request to repeat, or silence. Reproducing the caller's words
      is always 1.

cantonese_localisation — natural spoken Hong Kong Cantonese?
  10  Indistinguishable from a skilled Hong Kong agent: natural colloquial
      register, correct particles, natural code-switching where agents would.
  8   Natural spoken Cantonese with one or two slightly stiff phrases.
  6   Clearly Cantonese but noticeably formal, or written Chinese structures.
  4   Mostly written Chinese read aloud, or awkward mixed register.
  2   Largely not Cantonese (Mandarin phrasing, written-only forms).
  1   Not Cantonese, or unintelligible.
  Reusing Cantonese words lifted from the call is not evidence of fluency.

hallucination — did it invent things? (10 = nothing invented)
  10  Every specific (amount, date, term, coverage, procedure) is supported by
      the call, and the reply says so when the call does not give an answer.
  8   One harmless embellishment that changes nothing.
  6   One unsupported detail a caller might act on, but low risk.
  4   An unsupported promise, figure or procedure that could mislead.
  2   Several invented specifics.
  1   Invents coverage, an amount, a policy term or a procedure as fact. In an
      insurance line an invented limit is the expensive failure.

policy_accuracy — are the substantive facts of this call handled correctly?
  10  Every fact that mattered in this call is addressed correctly and
      completely.
  8   Correct, with one secondary fact incomplete.
  6   Partly right; something that mattered is incomplete.
  4   One substantive error, or the key fact missing.
  2   Mostly wrong.
  1   Wrong, or cites nothing where the call required it.

Return only the JSON object, with no commentary and no markdown fence. The note
is for auditors and never affects the score; keep it under 200 characters and
write it in English.

The human panel’s scorecard

Speech and conversation are judged by people from the Online Challenge, because naturalness is the one thing a machine cannot hear. To keep the judgement checkable, the panel scores each criterion separately against the same anchored scale, the pillar’s human score is the mean of the criteria, and every criterion is published on the team’s page. The anchors follow the ITU-T P.800 absolute category rating behind every MOS, with each step written as something a listener can check.

Speech criteria

  1. 1Reading accuracy. Every word, amount, date, policy or case number and English term read exactly as written, across all ten passages.
  2. 2Tones and pronunciation. Correct Cantonese tones and initials; no Mandarin-influenced or lazy readings; tone-sensitive pairs (報/保, 買/賣, 賠/陪) kept apart.
  3. 3Prosody and pacing. Phrasing, stress, pauses and breath that carry a 20-to-30-second passage; figures grouped the way an agent says them.
  4. 4Naturalness and voice quality. Sounds like a person speaking: no robotic timbre, glitches, clicks, clipping, noise or audible joins.
  5. 5Fit to the moment. Tone of voice suits what is said — care for bad news and complaints, clarity for instructions — in one consistent voice.

Conversation criteria

  1. 1Responsiveness. Replies promptly, without dead air, and without answering before the caller has finished.
  2. 2Turn-taking. Stops when the caller talks over it, picks up where it should, never talks over the caller.
  3. 3Task resolution. Handles what the caller needed, correctly and completely, inventing nothing.
  4. 4Conversational flow. Natural spoken Hong Kong Cantonese with a natural rhythm between turns, as a skilled agent would sound.

The scale every criterion is scored on

  1. 5Excellent. No fault a trained listener can find anywhere in the attempt. Indistinguishable from a skilled human agent; expected to be rare.
  2. 4Good. One or two minor faults that do not affect understanding or trust.
  3. 3Fair. Noticeable faults throughout; understandable, but clearly below a human agent.
  4. 2Poor. Frequent faults, or a single fault that changes meaning.
  5. 1Bad. Largely unintelligible, the wrong language, or mostly incorrect.

Caps

  • A misread amount, date, policy or case number caps reading accuracy at 2: a caller would act on the wrong figure.
  • A tone error that turns one word into another — insure into report, buy into sell — caps tones and pronunciation at 2.
  • A system that talks over the caller after being interrupted caps turn-taking at 2, however fast it is.

What a perfect 100 would take

Stated so nobody has to guess what the ceiling means. None of these is expected; each is what the arithmetic requires.

Recognition
Zero substitutions, deletions and insertions across all 30 real contact-centre calls in the attempt, with their crosstalk, hesitations and code-switching. The score is character accuracy, 100 × (1 − CER).
Understanding
A 10 on all four dimensions, from both models, in all three runs, for every one of 30 calls: 720 separate 10s, from a rubric that tells the judge 9 and 10 should be uncommon.
Speech
A 5 on all 5 criteria of the panel's scorecard, where 5 means no fault a trained listener can find in any of the ten passages, and a single misread figure caps reading accuracy at 2.
Conversation
A 5 on all 4 criteria of the conversation scorecard, where talking over a caller who interrupts caps turn-taking at 2.

The score card each team gets

Every team’s page publishes the headline of each pillar and every measurement behind the run that counts, so a team can see what it lost marks for. The board ranks only on the headline.

Ranked onAlso published
RecognitionCharacter error rate (CER), as 100 × (1 − CER)Mixed error rate (MER) · CER, script differences ignored · Errors from Simplified/Traditional script · Cantonese kept as Cantonese · Time to first response (median) · Time to first response (95th percentile)
UnderstandingJudge grade on the 1–10 rubric, as (grade − 1) / 9 × 100Answered the caller · Natural Hong Kong Cantonese · Nothing invented · Facts of the call right · Time to first response (median) · Time to first response (95th percentile)
SpeechHuman panel scorecard mean (1–5), as (mean − 1) / 4 × 100Panel: Reading accuracy · Panel: Tones and pronunciation · Panel: Prosody and pacing · Panel: Naturalness and voice quality · Panel: Fit to the moment · Machine pre-check · Clips delivered cleanly · Tones kept (買/賣 etc.) · Time to first response (median) · Time to first response (95th percentile)
ConversationHuman panel scorecard mean (1–5), blended with tail latencyPanel: Responsiveness · Panel: Turn-taking · Panel: Task resolution · Panel: Conversational flow · Machine-measured conversation · Time to first response (median) · Time to first response (95th percentile)

Aligned with how voice systems are evaluated

Nothing here is invented for this challenge. Each measure is the established method for its question, adapted to Cantonese and to insurance calls.

Character error rate
The standard for languages written without word boundaries, used by Cantonese and Mandarin recognition benchmarks; corpus-level, not a mean of per-clip rates.
Mixed error rate
Han characters counted singly and Latin words whole, as in Mandarin–English code-switching evaluations, so a misheard English word costs one unit.
Bootstrap intervals and rank bands
Percentile bootstrap over items (Bisani and Ney, 2004); entries whose intervals overlap are reported as tied, not ranked.
LLM-as-judge, single-answer grading 1 to 10
The MT-Bench convention (Zheng et al., 2023), with anchored bands, two models, three runs each and disagreement recorded.
Mean opinion score
ITU-T P.800 absolute category rating, the basis of the Blizzard Challenge and every published TTS listening test, scored per criterion.
Tail latency and barge-in
The 95th percentile time to first audio and whether the system yields when interrupted, the measures full-duplex dialogue benchmarks use.

Scoring weights

PillarWeightRequired
ASR — CER20%Every entrant
LLM — judge rubric, two models: 1–5 in Qualification; 1–10 with anchored bands from the Online Challenge25%Every entrant
TTS — Qualification: delivery 60% + tone survival 40%. Online Challenge: human panel scorecard (MOS) on ten 20-to-30-second insurance passages, machine pre-check shown but not counted30%Every entrant
E2E — latency 50% + yielding 30% + turns 20% until rated; Online Challenge: human judgement25%Optional

Core Score is ASR + LLM + TTS rescaled to 100 — hearing, understanding and speaking, the three the challenge is set on. Conversation is optional and sits outside it. Full Score covers all entered pillars and is withheld until every one of them is final — ranking a complete stack against a partially-evaluated one would be meaningless.

Character Error Rate

CER = (S + D + I) / |reference|

Computed corpus-level, weighted by reference length — not as a mean of per-clip rates. The two give different numbers: a mean over clips lets one short utterance weigh as heavily as a ninety-second call. Per-clip values are stored for diagnostics and never published as the headline.

Cantonese has no orthographic word boundaries, which makes Word Error Rate meaningless, so CER is the standard for this language. Mix Error Rate tokenises Han characters individually and Latin runs as whole words, so a misheard English word costs one unit rather than four characters — without it, code-switched speech is systematically over-penalised.

Run the scorer

The reference below is a real transcript from the sample split, folded the way the scorer folds it. Edit the second box — paste what your own system returned, if you have it — and the alignment is redrawn by the same functions the runner calls. Nothing is sent anywhere; this runs in your browser.

CER
0.0%
MER
0.0%
Substitutions
0
Deletions
0
Insertions
0
Reference length
40

Both rates agree on this text — there is no Latin run for them to disagree about.

  1. 嗯嗯
  2. 喂喂
  3. 午午
  4. 安安
  5. 你你
  6. 好好
  7. 哩哩
  8. 邊邊
  9. 保保
  10. 誠誠
  11. 保保
  12. 險險
  13. 客客
  14. 服服
  15. 務務
  16. 熱熱
  17. 線線
  18. 致致
  19. 電電
  20. 國國
  21. yy
  22. 你你
  23. 好好
  24. 哩哩
  25. 邊邊
  26. 保保
  27. 誠誠
  28. 致致
  29. 電電
  30. 過過
  31. 嚟嚟
  32. 㗎㗎
  33. 我我
  34. 叫叫
  35. cc
  36. oo
  37. rr
  38. ee
  39. 姓姓
  40. 楊楊
  • heard
  • substituted
  • dropped
  • added

From a measurement to a score

score = max(0, min(100, (1 − CER) × 100))

Every pillar reports its own unit — an error rate, a 1-to-5 rating — and the board shows one number, so each is put on a 0-to-100 scale. A rating maps straight across: 1 becomes 0, 5 becomes 100. An error rate is inverted, because lower is better: 0% becomes 100 and 100% becomes 0, linearly. The floor sits at a hundred per cent because that is what it means for nothing to have survived; it is not a threshold anyone chose. It was 60% until the corpus was measured against a real recogniser, which scored between 66% and 100% on it — so every entrant floored and the pillar separated nobody. On the board each pillar is headed by this 0-to-100 score, higher is better, the same direction as the Core Score, with the raw measurement shown beneath it in its own units — so a low character error rate and a high score sit together and neither can be misread for the other.

The floor has moved twice, and figures either side of a move are on different scales: 20% until August 2026, then 60%, and a hundred per cent since the corpus was measured against a real recogniser. Every published row records the scoring version that produced it, which is how you tell which scale a number is on.

What a machine decides, and what a person decides

Every figure that gates qualification is measured by machine. Recognition is a character error rate. Understanding is an LLM judge over a published rubric, run several times with disagreement recorded. Speech is delivery — audio arrived, right length, sane level, unclipped, inside the deadline — at 60%, and tone survival at 40%: the synthesised audio is fed back through a recogniser and we check whether the six tones are still distinguishable. Conversation is time to first byte and whether the system yields when interrupted.

People rate two things and only two: naturalness for conversation, and MOS for speech. Those are opinions and they are labelled as opinions. A qualification round can have more entrants than a panel can listen to, so nothing that decides who continues waits on a human. Where a pillar is published on its machine measurement, the board says “measured, not rated” next to the number rather than letting a reader assume a person sat with it. From the Online Challenge the standard changes: speech and conversation count only on a human judgement, entered by the organisers per team; until it is in, the pillar reads “Waiting for human judgement” with the machine pre-check shown and not counted, and the Core Score shows the points earned so far and how many are still to be added. Understanding is judged on a stricter 1–10 rubric with anchored bands, and its four dimensions are published on each team’s page. The test also rotates from the Online Challenge: every live scored attempt at recognition and understanding draws its own calls (a batch upload sits the round’s fixed paper) from the round’s pool, in the same easy, normal and hard mix, with calls the team has not heard before coming first; every speech attempt reads one 20-to-30-second passage from each of ten insurance topics, out of six per topic (sixty in all), and none of a team’s first six speech runs, practice and scored counted together, repeats a passage; scored runs stay at three per pillar.

One automatic rule is worth stating because it caps a score rather than computing one. If none of the six tones can be recovered from what a synthesiser returned, speech is capped at 3 of 5 however well it delivered. Delivery proves the audio is the right shape, not that it is speech: a 180 Hz sine wave of plausible length passes every delivery check, and without the cap it would score full marks on a pillar worth 30% of the Core Score. The cap is the difference between “we could not verify this is Cantonese” and “every tone correct”.

Normalization spec v1.5.0

Undocumented normalization is the single largest source of ASR benchmark disputes. The exact code below runs in our scorer and is published for participants to run locally, so both sides compute the same number from the same bytes.

RuleCanonical form
ScriptTraditional, and the two sides are treated differently on purpose. YOUR transcript is not folded: Simplified is different output and is charged, and the run page shows how much of your error it was. The REFERENCE is folded to Hong Kong Traditional with OpenCC (cn→hk) before scoring, because some delivered references mix Simplified in and our corpus’s orthography is not your mistake. Cantonese-specific characters are held out of that fold — OpenCC rewrites 嚇, 晒, 搵 and 戶, which would corrupt the answer key it is meant to clean.
LanguageEvery clip is Hong Kong Cantonese. Calls found to be Putonghua are relabelled and leave every draw, the download and the published counts — you will not be scored on Mandarin audio. English appears throughout as code-switching, which is the point of Mix Error Rate: a misheard English word costs one unit rather than four characters.
SpellingFrom spec 1.4.0, common Cantonese spelling twins count as the same word on both sides: 番/返, 唔洗/唔使, 既/嘅, 野/嘢, 黎/嚟, 左/咗, 我地/我哋, 果個/嗰個, d/啲 and a few more. Different words stay different: 的 against 嘅 or 沒有 against 冇 is still an error, because that is translation.
Number glossesSome delivered references write a spoken number and then its digits in brackets, such as 三六五六(3656). From spec 1.5.0 the bracketed digits are ignored on both sides, so a number counts once, as said. Do not output the bracketed digits; write numbers in Chinese or in Arabic, both score the same.
Line orderSome delivered references have lines out of order against the audio. From 8 Oct 2026 the reference’s lines are also scored in the order your transcript has them, and the lower error count is kept. Only the order of whole lines moves, never the words, so this can lower a charge and never raise one.
WidthNFKC; full-width folds to half-width
CaseLatin lowercased
PunctuationRemoved, CJK and Latin
WhitespaceRemoved entirely
NumeralsChinese folded to Arabic — 三百 = 300
CurrencySymbol and 蚊 / 元 folded away
FillersRetained. They are speech.
Background speechNot transcribed. The reference is what the caller and the agent say; a television, a radio or anyone else in the room is not part of the call and is not in the answer key. A system that filters it out is right to.

Intervals and rank bands

Every number is accompanied by a 10,000-sample percentile bootstrap, resampled over clips. Entries whose 95% intervals overlap are placed in a shared rank band and reported as tied. Publishing a strict order on differences inside the noise floor is false precision, and it is the failure every serious benchmark is eventually criticised for.

The protocol

Participants implement a WebSocket server; our orchestrator is the client. Protocol v1.2, audio as pcm_s16le_16k. The per-clip deadline scales with clip length — a 60-second clip allows 30000ms from end-of-utterance to first response byte, measured on our monotonic clock only.

That deadline is the real integrity control. We do not claim audio is undownloadable — it is streamed to your process and that claim would be unenforceable. What we claim is narrower and true: the response window is far too short for a human to transcribe the audio, every copy of a dataset pack is issued to a named recipient and recorded in the audit log, the clip subset rotates between windows, and final standing is verified against a held-out set nobody has seen.

We do not watermark the audio. A per-entrant signal in the waveform would either be audible enough to change what is being measured or weak enough to survive nothing, and a control that cannot be relied on should not be described as one. Attribution rests on the issue record instead, which is enforceable because it names a person.

The conversation pillar

The other pillars measure an answer. This one measures behaviour between turns, so it runs a different protocol: we hold three scripted conversations against the entrant’s endpoint and time the gaps.

Latency is the interval from our end-of-utterance to the first frame announcing that the system has begun speaking — not to a completed reply, so buffering the whole answer before announcing it cannot look fast. We score the 95th percentile rather than the median: a system that is usually quick and occasionally takes three seconds is experienced as a slow one. Below 300ms the score is flat, because a caller cannot tell 120ms from 280ms and rewarding that difference optimises something nobody experiences. It reaches zero at 2000ms.

Yielding the floor. Every scenario ends on a turn where the caller talks over the reply. We measure how long the system keeps speaking afterwards and count it as yielding within 500ms. A system that is fast and will not stop talking is worse than a slow one, and a benchmark that measured only latency would rank it first.

Once a panel has rated the transcripts the pillar is 40% latency and 60% human-rated naturalness. Until then it publishes on the machine measure — 50% latency, 30% yielding the floor, 20% turns completed — and the board marks that number “measured, not rated”, so a fast system is never read as a good conversationalist before anyone has checked. In the Online Challenge the rated half is the panel’s four-criterion scorecard, and until the panel has scored a team the pillar reads “waiting for human judgement”: the machine measure is shown and not counted, and once the scorecard is in the pillar is 40% latency and 60% scorecard.

Rotation and exposure

Each scored run draws a fresh random subset from the rotation pool, stratified across difficulty tiers so a lucky draw of short clips cannot flatter a system. The draw is seeded per window and recorded, so any run can be reconstructed exactly.

Repeated runs still accumulate exposure, and the arithmetic is not favourable: with a pool of 56 and a draw of 30, an entrant who uses every run in their quota has seen most of the pool. That is why the pool grows between editions and why final standing is confirmed against a held-out set of 11 clips that no entrant ever receives.

Where this sits

Cantonese speech evaluation is not a blank field, and it would be false to call this the first Cantonese benchmark. Several test sets already exist and predate this one:

WSYue-eval
Ships with WenetSpeech-Yue, a 21,800-hour Cantonese corpus. Covers both recognition (short and long utterances, code-switching, varied acoustics) and speech synthesis.
MDCC
Multi-Domain Cantonese Corpus, an established read and spontaneous speech test set.
Common Voice zh-HK
Crowd-recorded, widely used as a recognition baseline.
HKCanto-Eval
Cantonese language understanding and cultural comprehension.
CantoNLU
Cantonese natural-language understanding tasks.

What is new here is narrower and, we think, more useful:

Real enterprise call audio
Consented Hong Kong insurance contact-centre recordings, with the hesitation, crosstalk and background of an open floor. Read speech is a different problem, and doing well on it does not predict doing well on this.
Production systems, not checkpoints
The existing sets score models. This one scores the services a company could actually buy, over a connection we open to a server they run.
End-to-end conversation
Latency, turn-taking and yielding the floor, measured on our clock. No existing Cantonese set covers this.

Because a number means nothing in isolation, entrants are cross-anchored against WSYue-ASR-eval and both figures are published. A 12% character error rate here alongside 6% on WSYue tells a reader exactly how hard this audio is; either number on its own tells them nothing.