Cyberport × Prudential · Cantonese Voice Benchmark 2026

The entrant's guide

What is measured, when it happens, and how to build the endpoint we connect to. Everything here is generated from the platform itself, so a number in this guide is the number the platform is using.

Issued 24 September 2026 · Protocol v1.2 · Live version at https://cvc-benchmark.pubrio.com/integration

Step by step, with every screen you will meet: https://cvc-benchmark.pubrio.com/guide

• 1. What this measures
• 2. The dates
• 3. How a score is made
• 4. Building the endpoint
• 5. Practising
• 6. What you agree to
• 7. Getting help

1. What this measures

Production Cantonese voice systems, on real Hong Kong insurance calls — not read speech, not a scripted corpus. Background noise, people talking over each other, callers who are upset. That is the point: a system that scores well on clean audio and badly here is a system that will disappoint a contact centre.

Four pillars. You may enter as few or as many as you like; the three core pillars are what the programme is about, and conversation is optional. You say which in your application, sent from your portal: it is approved the moment you send it, and it is the one step between practice and your first scored run.

Every published figure carries a confidence interval, and two systems whose intervals overlap are reported as tied rather than ranked. A benchmark that ranks noise is worse than one that admits it.

2. The dates

All times are Hong Kong time. The qualification round decides who continues; the online challenge produces the published standing. The two draw from separate pools, so a clip you were qualified on is never a clip you are scored on.

3. How a score is made

A scored run draws 30 clips at random from the round's pool, stratified by difficulty — roughly a fifth easy, two fifths hard, the rest normal — so nobody gets an easy paper. You get 3 scored runs per round in each pillar your application enters. A run we break costs you nothing.

Recognition

Character error rate: substitutions plus deletions plus insertions, over reference characters, at corpus level. Cantonese has no word boundaries, so word error rate would be meaningless. Both your output and the reference pass through the published normalisation spec first — read it at https://cvc-benchmark.pubrio.com/methodology before assuming a difference is an error. Most surprises on first contact are normalisation, not recognition.

The reference is what the caller and the agent say. Background speech — a television, a radio, anyone else in the room — is not part of the call and is not in the answer key, so a system that filters it out is doing the right thing. Fillers the two speakers say are kept: they are speech.

The deadline

Half the clip's length, with a floor of 3 seconds: max(3000, duration_ms × 0.5). It is measured on our clock, from the end of the audio to your first response byte; our own connect and handshake time is measured separately and never counted against you. Read it from deadline_ms in the handshake rather than computing it — a round may relax the factor.

What is published

Your number, its interval, and the per-pillar breakdown. Nothing is published under your name unless you ask for it: an entrant may compete under a codename and decide after seeing the result.

4. Building the endpoint

You run a WebSocket server. We connect to it as a client — you never call us. The URL must be wss:// with a certificate that verifies against the public roots; a self-signed or expired one fails the connection test. We send no headers and no token, so if you want a shared secret, put it in the URL you register.

1. We open the socket and send hello, naming the pillar, the clip's length and format, and the deadline.

   {"type":"hello","protocol":"1.2","run_item_id":"3f2a…","pillar":"asr","clip":{"sample_rate":16000,"encoding":"pcm_s16le_16k","duration_ms":18000},"deadline_ms":9000,"formats":["pcm_s16le_16k"]}

2. You answer ready. This is not optional and it is the step people miss — nothing is streamed until it arrives, and an endpoint that connects and stays silent is failed as an incomplete handshake.

   {"type":"ready","accept":"pcm_s16le_16k"}

3. We stream the audio as binary frames — real WebSocket binary, not base64: 3200 bytes each, one every 100ms, 16 kHz mono signed 16-bit PCM. The WAV header is stripped before we send, so there is no RIFF to look for.
4. We send end_of_utterance. Your deadline starts here — not at hello, and not at the first byte of audio.

   {"type":"end_of_utterance"}

5. You send the transcript. The first one settles the clip.

   {"type":"transcript","text":"你好，我想claim返個醫療費"}

Speech runs the other way — we send text in hello and no audio, and your latency is measured at your speaking frame rather than at the end. Conversation sends numbered turn frames on one socket; answer after end_of_utterance and carry the same index back, because while the caller's turn is playing nothing is listening yet.

A server that already works

This runs as it stands and passes the connection test with a stub model. Replace one function. The same file, plus a Node version and a Dockerfile, is in the starter kit at https://cvc-benchmark.pubrio.com/api/v1/starter-kit.

# A Cantonese Voice Benchmark endpoint, in about forty lines.
# python -m pip install websockets
# python server.py      →  ws://0.0.0.0:8080
#
# Put your recogniser in transcribe(). Everything else is the protocol.
import asyncio, json, os
from websockets.asyncio.server import serve

CHUNK = 3200  # bytes per frame we receive: 100ms of 16kHz mono s16le


def transcribe(pcm: bytes) -> str:
    """Raw 16kHz mono signed 16-bit PCM. No WAV header — we strip it."""
    return "你好"  # ← your model here


async def handle(ws):
    audio = bytearray()
    async for message in ws:
        # Binary frames are the caller's voice, arriving while they speak.
        if isinstance(message, (bytes, bytearray)):
            audio.extend(message)
            continue

        frame = json.loads(message)

        if frame["type"] == "hello":
            # Nothing is streamed until you answer this. An endpoint that
            # stays silent here is failed as an incomplete handshake.
            await ws.send(json.dumps({"type": "ready", "accept": "pcm_s16le_16k"}))

        elif frame["type"] == "end_of_utterance":
            # The deadline clock starts at this frame, not at hello.
            text = transcribe(bytes(audio))
            await ws.send(json.dumps({"type": "transcript", "text": text}))
            return  # one clip per connection


async def main():
    async with serve(handle, "0.0.0.0", int(os.environ.get("PORT", 8080))):
        await asyncio.get_running_loop().create_future()

asyncio.run(main())

The full protocol, including every failure and the sentence we will show you for it, is at https://cvc-benchmark.pubrio.com/integration.

5. Practising

Two things cost you nothing and both are in your portal.

The connection test. Three real clips, about ten seconds, at the concurrency you declared, with the same pacing and timeouts a scored run uses. If it fails we automatically run the same three against our own reference endpoint, so the result tells you whether the fault is yours or ours rather than leaving you to guess.

The practice pack. 8 recorded calls with their reference transcripts, so you can compute your own error rate before booking a scored window. Each clip ships with the measurements behind our claim that it is a recording rather than a synthesiser — median frame level, noise floor, and runs of digital silence — so you can check that claim instead of taking it. Your score never comes from these clips.

The audio is one mono track at 16 kHz. Both sides of the call are mixed into it, which is exactly what your endpoint receives, so a quiet stretch is the other person listening rather than audio that is missing.

6. What you agree to

The dataset licence is the part that matters: this edition only, no redistribution, no training on it. The recordings are real customers' voices and they sit behind an account for that reason.

Your model stays yours. We publish numbers, not systems, and we never ask for weights. A run we break is re-run at no cost to your quota, and every failure is attributed — ours, yours, or the network — with the per-item evidence behind it. If you disagree with a number, open a dispute from the run page; it is answered with that evidence.

Full terms at https://cvc-benchmark.pubrio.com/rules and https://cvc-benchmark.pubrio.com/terms.

7. Getting help

An error message that did not make sense to you is a bug in our writing rather than a question you should have known the answer to. Tell us and we will fix the page.