Cantonese Voice Benchmark粵語語音基準

Cantonese Voice Benchmark 2026.1

How they did it.

Every entrant writes up their system before their Full Score is released. These are the ones whose teams chose to publish. A rank tells you who came first; this tells you what to try.

Edition
Cantonese Voice Benchmark 2026.1
Published write-ups
5
Full leaderboard
Full leaderboard
  1. LoomWerkcommitted scores
    19.6%Hearing
    8.82Understanding
    5.00Speaking
    published 08 Oct, 00:54 HKT
    01 — What the stack is Four models sit in the path. All are hosted APIs; none is ours. Recognition runs iFlytek 語音聽寫大模型 (spark_asr_v2) and Microsoft Azure Speech in parallel, both over the whole clip at end-of-utterance, with the better transcript taken. Understanding runs an OpenRouter chain: mistralai/mistral-small-2603, then amazon/nova-micro-v1, then deepseek/deepseek-v4-flash. Speech synthesis is cantonese.ai, with Microsoft Edge TTS zh-HK-WanLungNeural as the deadline fallback. Text is normalised with OpenCC (s2hk) plus our own insurance glossary. The endpoint itself is a WebSocket server we wrote ourselves in Bun and TypeScript (protocol v1.2), one connection per clip, one endpoint per pillar, public at wss://cvc.cdc-hk.com. 02 — What was trained, and on what No model was fine-tuned. No adapters, no training runs. Three things were built by hand: 1).A Prudential insurance glossary — 187 canonical terms and 101 observed mis-renderings mapped onto them, taken from prudential.com.hk policy wording. iFlytek's own hotword mechanism was measured inert on our application, so the glossary is applied to the finished transcript instead, longest alias first, so one rewrite cannot cascade into another. 2).Traditional-Chinese conversion that protects the four characters OpenCC's s2hk fold corrupts (嚇 晒 搵 戶). 3).System prompts and answer rules for the LLM pillar, tuned only against the organisers' published practice pack — 8 clips with reference transcripts. Scored audio was never used for tuning, and no public Cantonese corpus was used. 03 — What runs at inference At end_of_utterance the clip is buffered as PCM and both recognisers are run on the whole of it. iFlytek is called in 10-second windows, all issued in parallel and reassembled in order, capped at 20 windows, with a retry on any window that comes back empty. Azure is split so that each part aims for at least four parts and never more than six, because eight parallel parts made the resource silently drop the last two. Transcript pieces are accumulated rather than overwritten, and the first engine to return usable text wins. Output then passes OpenCC s2hk, then the glossary, then goes back to the platform. The LLM runs at temperature 0.3, maximum 220 tokens, with reasoning disabled, and the first complete answer in the chain wins — a model that cuts off mid-sentence is discarded. Speech is streamed rather than buffered, and whichever engine produces first audio inside the grace period keeps the line. No retrieval, no reranking, no ensemble, no speaker labels, no timestamps. 04 — Where the humans are No human is inside a scored run. Human work sits before it: authoring the glossary, the prompts and the correction rules; reading the practice pack's reference transcripts; watching runs and fixing code between them. A scored run is started by the platform, answers on the same code path every time, and is never edited afterwards. 05 — Hardware and latency The endpoint runs on our own normal cloud host — 4 vCPU, 32 GB RAM, no GPU — reached through a named Cloudflare Tunnel, so Cloudflare terminates TLS and no inference happens there. All measured latency is model time. Recognition: 1.5 s after end-of-utterance over the public tunnel on a 25-second clip, and 12.9 s on the platform's own scored run of 11 real clips against a 4.3–15.6 s deadline. LLM: 3.2–4.7 s against a 4.3 s deadline. Speech first audio: 1.8–2.6 s against a flat 3.0 s clock, held inside it by a 2.5 s grace on the cantonese.ai voice. The scored recognition run stands at 22.6% CER. 06 — Known weaknesses iFlytek's response time for the same clip swings from 0.9 s to over 30 s, and it finalises empty under load — which is why Azure runs in parallel, and why we let the service idle before a scored run. Azure truncates silently when a part opens deep inside silence. The LLM pillar is our weakest: answers can take 14 s or more on long calls, at or past the deadline, and a short answer loses detail. Speech is voice-level, not sentence-level: prosody is the vendor's, and numbers and English terms are read as written rather than re-normalised. Recognition errors concentrate on code-switched proper nouns and on Cantonese particles that both engines write as Mandarin-ish characters. Next: pre-warm both engines at connection open, split long calls into sub-deadline LLM segments, and expand the glossary from the scored error log.
  2. 聴港committed scores
    16.4%Hearing
    8.92Understanding
    5.00Speaking
    published 30 Sept, 19:53 HKT
    This project is a Cantonese Voice AI endpoint designed for Hong Kong insurance customer service. It processes telephone conversations and provides automatic speech recognition, conversation understanding, insurance-domain question answering, and text-to-speech responses. The system runs on a rented host with two NVIDIA L20N GPUs, each providing 72 GB of GPU memory. The architecture separates speech recognition, language understanding, and speech synthesis so that the services can be tested independently and measured under realistic customer-service workloads. GPU 0 runs Qwen3-ASR 1.7B and SenseVoice-Small for Cantonese speech recognition. The ASR adapter is exposed on port 9882. It accepts WAV or PCM audio and can be connected to the gateway through HTTP or Socket.IO. The ASR pipeline supports voice activity detection, audio sample-rate normalization, mono conversion, OpenCC conversion to Hong Kong Traditional Chinese, insurance-domain hotwords, Cantonese-English code-switching, English term preservation, repeated-word analysis, same-sound error analysis, and safe post-processing. Important insurance and customer-service terms such as policy, claim, premium, processing, conference, confirm, Bank confirm, account, app, email, link, check, and check check should remain in English when they are actually spoken. The system keeps raw transcription separately from score text and display text so that post-processing does not hide the model’s original recognition result. SenseVoice-Small is used as an additional ASR model for comparison. It analyzes audio quality, including sample rate, channels, speech ratio, estimated SNR, noise RMS, and background-speech risk. When the customer or agent speaks quietly, the preprocessing stage can apply voice boosting, dynamic compression, high-pass filtering, low-pass filtering, and noise reduction. When a recording has a high risk of television or continuous background speech, the system can select a stronger adaptive enhancement profile. This feature is a heuristic background-noise analysis and not a complete source-separation model. If television speech overlaps with the real customer or agent, a single-channel telephone recording cannot guarantee perfect separation. GPU 1 runs Qwen3-32B-FP8 as the language model. The LLM endpoint is exposed on port 30000. It is instructed to answer in natural, concise, and professional Hong Kong Cantonese using Traditional Chinese. It should preserve necessary English insurance terminology, avoid Mandarin-style wording where possible, and never output hidden reasoning, analysis, invented facts, unnecessary disclaimers, or information that is not present in the call. When the transcript does not contain enough information, the model must state the limitation instead of guessing policy details, payment dates, customer identity, or company procedures. GPU 0 also runs Fish Audio S2 Pro for Cantonese text-to-speech. The TTS API is exposed on port 8104 through the SGLang-Omni backend. It supports reference audio, reference text, reference voice conditioning, streaming output, Cantonese speech synthesis, and up to four concurrent requests. TTS evaluation includes time to first audio, total generation time, voice consistency, tone preservation, naturalness, and concurrency stability. The system is evaluated with the Cantonese Voice Benchmark. Evaluation covers character error rate, missing words, wrong characters, homophones, repeated words, hallucinated text, English recognition, background-noise handling, quiet-speaker handling, end-to-end latency, TTS time to first audio, voice similarity, and LLM answer accuracy. Benchmark recordings are kept as holdout data and are never used for training, validation, prompt tuning, or parameter tuning. Raw transcripts, normalized score text, display text, replacement decisions, audio-quality metadata, and benchmark results are retained for audit and error analysis. The main current limitation is overlapping television speech in single-channel audio; stronger denoising may also remove Cantonese endings or English consonants, so every enhancement must be validated against the fixed benchmark.
  3. Fathom10competing under a codenamecommitted scores
    16.8%Hearing
    8.63Understanding
    5.00Speaking
    published 30 Sept, 17:54 HKT
    # System description — Cyberport × Prudential benchmark ## 01 — What the stack is Everything runs behind one Python WebSocket server (`websockets` 17.x) with one endpoint per pillar on the same host. Recognition is **Qwen3-ASR-1.7B** (Qwen/Qwen3-ASR-1.7B, via the `qwen-asr` inference library) running in bfloat16 on a local CUDA GPU. Reasoning uses **Qwen3.5-9B** (kwangsuklee/Qwen3.5-9B-Claude-4.6-Opus-Reasoning-Distilled-GGUF) served locally by **Ollama** — no hosted LLM APIs. Speech synthesis uses Microsoft's **Edge TTS** hosted neural service (voice `zh-HK-HiuMaanNeural`), converted to 16 kHz PCM with **ffmpeg**. Text normalisation uses **OpenCC** (s2hk). We trained none of these models ourselves. ## 02 — What was trained, and on what Nothing was fine-tuned — no adapters, no training runs. What we did add: - **Prompt engineering**: a ten-term Prudential call-centre vocabulary passed as decoding context to the ASR model, and Cantonese system prompts for the reasoning and conversation replies. - **Rule-based post-processing**, derived mechanically (confusion matrix) from the published practice sample pack only — its reference transcripts, ~8 clips. Examples: mapping engine spelling variants to the references' orthography (翻→番, 俾→畀, 呢度→哩度), 户→戶, and guarding 喂 from OpenCC conversion. No public Cantonese corpora were used for training or adaptation; there was no time budget for fine-tuning before the deadline. ## 03 — What runs at inference - **ASR**: audio buffered until `end_of_utterance`, then split at the lowest-energy 100 ms frame inside each 30 s window (searching the middle 60%); all segments batched (max batch 8) in one transcribe call, `language="Cantonese"`, bfloat16, max_new_tokens 4096, with the insurance-vocabulary context prompt. Post-processing: OpenCC s2hk → traditional HK, plus the spelling-variant map above. - **Reasoning**: transcript + task into Ollama chat, temperature 0, ≤400 new tokens, answer returned in a `transcript` frame per protocol 1.2. - **Conversation**: no ASR — replies are generated from the turn's provided text (the pillar doesn't ask for a second recogniser), last 8 conversation messages kept, temperature 0, ≤120 tokens, then synthesised with the same TTS voice. - **Speech/Conversation audio**: Edge TTS mp3 → ffmpeg → 16 kHz mono s16le, streamed in 3200-byte frames; `speaking` is sent before synthesis so time-to-first-byte is minimal; `barge_in` cancels playback and yields the floor immediately. - No retrieval, no reranking, no ensemble. ## 04 — Where the humans are None inside a scored run — the full path is automated, one clip per connection, no human review of outputs. Outside scored runs, humans (the authors) wrote the prompts and post-processing rules using the published practice pack, and watched server logs during practice runs. No human labelling of benchmark audio, no human-in-the-loop at inference time. ## 05 — Hardware and latency One RunPod Secure Cloud pod (Singapore region) with a single **NVIDIA RTX PRO 4500 Blackwell (32 GB)**; the ASR model (4.4 GB) and the Ollama model (5.6 GB) both live on the same GPU. Clients reach it through the RunPod TLS proxy. Measured on your clock in scored runs: speech pillar **0.27 s median / 0.28 s p95**; reasoning **10.94 s median / 21.72 s p95** against deadlines of roughly half the clip length (worst observed: 38.4 s of a ~93 s budget). The reasoning latency is dominated by batch ASR of the full clip, not the LLM. ## 06 — Known weaknesses - **Noisy audio**: recognition degrades sharply on low-SNR clips — on one heavily degraded practice clip CER was ~0.80 against ~0.25 elsewhere. A denoiser front-end (e.g. DeepFilterNet) is the first thing we'd add. - **Orthography over-fitting**: the spelling-variant map is fitted to the practice pack's reference style; if a scored round uses different transcription conventions it could cost a couple of CER points. - **Reasoning latency**: 11–22 s because ASR runs only after `end_of_utterance`. Streaming/chunked ASR would cut this to near speech-pillar numbers. - **Conversation content**: replies are deliberately short and safe (grounded, no invention); prosody is fixed — one neural voice, no emotion or pausing, and barge-in cuts audio abruptly mid-utterance. Fine-tuning the ASR on real call-centre Cantonese and a more expressive TTS voice are the obvious next steps.
  4. CantoFlowcommitted scores
    21.9%Hearing
    published 28 Sept, 22:20 HKT
    CantoFlow — Cantonese speech recognition for Hong Kong calls Entered this round: Recognition only. 01 WHAT THE STACK IS - Recognition model: Qwen3-ASR-0.6B by Qwen (open weights, Hugging Face Qwen/Qwen3-ASR-0.6B, revision 5eb14417), converted locally to an 8-bit MLX checkpoint (group size 64). We did not train it. - Runtime: mlx-qwen3-asr 0.3.5 on Apple MLX 0.31.2, inside CantoFlow 0.2.4 (build 20260822.0038), our Cantonese speech-to-text app for macOS, in local mode. - Endpoint: our own WebSocket server (Python, protocol v1.2, pcm_s16le_16k), reached through a Cloudflare tunnel. It sends each clip to CantoFlow's local HTTP API (POST /v1/audio/transcriptions) once. - Fallback: if CantoFlow returns no text, the endpoint runs OpenAI Whisper large-v3-turbo locally (openai-whisper 20250625). It was not triggered in any platform run. - Script conversion: OpenCC (opencc-python-reimplemented 0.1.7), a rule-based converter, not a model. - No cloud model or cloud inference API is used for recognition; Cloudflare is used only to expose the public endpoint. 02 WHAT WAS TRAINED, AND ON WHAT - We performed no additional training: no fine-tuning and no adapters. CantoFlow's installer downloads the released Qwen3-ASR-0.6B weights and quantises them to 8-bit. The only task-specific context supplied at inference is the prompt. - The submitted system does not supply practice or evaluation audio, transcripts, personal identifiers, per-call facts, or utterance-specific phrases to the model. The frozen personal vocabulary contains only 10 public insurance-domain terms and excludes pack-derived names, numbers, dates, and phrases. The practice pack was used to evaluate the system. 03 WHAT RUNS AT INFERENCE - On hello the endpoint answers ready at once. The clip's audio frames are buffered and, after end_of_utterance, sent to CantoFlow as one 16 kHz mono WAV. - CantoFlow runs one transcription at a time. The endpoint declares concurrency 1 and does not claim parallel ASR capacity. - Segmentation: audio longer than 30 s is split at low-energy points into segments of at most 30 s; each segment is decoded separately and the texts are joined. - Decoding: greedy (temperature 0; no beam search, no speculative decoding), float16 activations, a duration-aware cap on new tokens per segment, and a stop on detected repetition. Language is auto-detected (see 06). No timestamps, diarization or forced alignment. No retrieval. - Prompt: the prompt contains a fixed Cantonese/Traditional-Chinese instruction, the frozen 10-term public insurance vocabulary (保誠, 保誠保險, 保單, 保單持有人, 保費, 繳費, 賠償, 理賠, 客戶服務, 致電), and generic built-in Hong Kong vocabulary terms until the 500-character limit is reached. - Post-processing: CantoFlow converts its output with OpenCC s2hk. The endpoint then applies constrained output formatting before emitting the transcript (OpenCC s2t for remaining Simplified characters, keeping Cantonese forms 吓, 晒, 搵, 戶 and 喂); it does not apply the platform's scoring normalisation. Nothing else changes the text. 04 WHERE THE HUMANS ARE - No human transcribes, edits, or selects individual benchmark outputs. Every submitted transcript is produced by the system alone. - Outside scored runs, people wrote the endpoint code, the prompt instruction and the 10-term vocabulary, ran practice runs to measure error rates, and watched endpoint logs during runs for failures. No output was changed. 05 HARDWARE AND LATENCY - One Apple M4 Mac mini (4 performance and 6 efficiency cores, 16 GB memory, macOS 26.5.1) in San Po Kong, Hong Kong. The endpoint and CantoFlow run on the same machine. - On the first scored run, recognition took 7.8 to 27.6 s per clip for clips of 74 to 312 s. The platform measured a median response of 10.7 s and a 95th percentile of 15.0 s. - Measured on the platform: practice run, 8 clips, CER 30.1%; first scored run, 11 clips, CER 24.5%, every item inside its deadline. 06 KNOWN WEAKNESSES - Both speakers are mixed in one channel. Short backchannel turns, speaker changes, and English code-switched words remain challenging conditions. - Output keeps the engine's own Cantonese spelling (e.g. 啊 where references write 呀); no spelling rules were added for this round. - What we would fix next: the endpoint sends the language hint (yue) in a form CantoFlow ignores, so the engine's default applies. We would pass it correctly and re-measure on a practice run before any further scored run.
  5. Yardarm7competing under a codenamecommitted scores
    18.5%Hearing
    8.90Understanding
    4.68Speaking
    published 23 Sept, 18:16 HKT
    NotOnlyOffice enters Cantonese Voice Benchmark 2026.1 with the three Core pillars that define the published Core Score: speech recognition (ASR), call understanding (LLM), and Cantonese speech synthesis (TTS). We are not entering the optional end-to-end conversation pillar in the qualification round. Our submission is built for the benchmark’s stated setting—real Hong Kong insurance contact-centre calls on a single mixed mono channel, with background noise, overlap, code-switching, and spoken Hong Kong Cantonese rather than clean read speech. We implement the organiser’s WebSocket protocol (version 1.2) as a service we host; their orchestrator connects to us as the client. For portal registration we provide three wss:// URLs on one TLS host, using separate paths for ASR, LLM, and TTS, all served by one integrated server codebase with shared connection handling and logging. Each evaluation clip uses one connection only; we do not carry state across clips. We respond to hello with ready before any audio is streamed, keep the socket limited to protocol frames, and route work according to the pillar field and registered endpoint. For ASR and LLM, the organiser sends paced 16 kHz mono PCM in 100 ms binary frames after ready, then end_of_utterance. We must return a single transcript JSON frame before deadline_ms, which we read from hello. We use an internal time budget with margin before the hard deadline so clips rarely fail for lateness. ASR output is one plain text string in Hong Kong Traditional, reflecting spoken Cantonese as heard on the call. We transcribe both parties in chronological order, including overlapping speech, and we retain fillers such as 呃 and 嗯 because they are scored as speech. We do not add speaker labels or editorial tags. In scored mode we avoid aggressive cleanup that would drop short or quiet segments, since deletions count against character error rate. After recognition we apply OpenCC simplified-to-Hong-Kong-traditional conversion with protection for Cantonese-specific characters that generic conversion would distort, plus conservative repetition and hallucination suppression that does not remove genuine utterances. The LLM pillar receives the same audio stream. The task string in hello is passed unchanged into our prompt. Architecturally we use a cascade: streaming ASR produces a transcript, then a large language model generates the required reply. The response must answer the caller in natural Hong Kong Cantonese, grounded only in what the call contains—not a recap of the transcript. We aim for concise answers that still surface facts downstream processes care about: policy numbers, claim amounts, dates, medical terms, and what was agreed or promised, using Arabic numerals where those facts are spoken. When the call does not establish something, we state that limitation rather than invent coverage or amounts. Before sending, we check that the answer is not unduly echoing the transcript. For TTS, hello carries the text to speak and no input audio. The deadline starts at hello. After ready, we emit speaking as soon as the first synthesized audio byte exists, stream 16 kHz mono signed 16-bit PCM without a WAV header in fixed-size chunks, then send turn_done. For scored runs we use hosted and/or self-hosted components declared here: primary ASR via Alibaba DashScope Qwen3-ASR-Flash and/or self-hosted SenseVoice-small-yue; an optional second-pass recognizer on long clips when time allows (e.g. Qwen3-ASR-1.7B or GLM-ASR-Nano); LLM via Anthropic Claude (faster models on the shortest clips, stronger models when the deadline permits); TTS via a Cantonese neural service such as Azure zh-HK Neural or ElevenLabs with a fixed voice profile. TLS is terminated with publicly trusted certificates; compute is deployed in a nearby region (e.g. Hong Kong, Singapore, or Tokyo) with modest declared concurrency per pillar URL. We do not train, fine-tune, or adapt on benchmark evaluation or practice audio. We may use public insurance wording in prompts or hotwords for terminology only, never to assert facts absent from the call. Audio is processed solely to produce submissions and is not redistributed. Scored runs are fully automated with no human transcription or editing. Remaining risks include overlapped speech, digit and policy-number errors propagating from ASR into understanding, and TTS tone survival under the benchmark’s automatic checks.