Cantonese Voice Benchmark粵語語音基準
EN繁中

接入

點樣起你個端點。

你架一個 WebSocket 伺服器,我哋做客戶端駁上去。呢一版就係成套通訊協定——兩個方向嘅每一個訊框,加埋一個貼咗就行得嘅伺服器。語音支柱一般兩至三個鐘搞掂。

整體係咁

我哋係客戶端。你唔使打嚟我哋度:你自己架伺服器、登記個網址,測試開始嗰陣我哋會駁上去。咁樣我哋先可以喺我哋揀嘅條件下量你個系統,而唔係喺你揀嘅條件下。

個網址一定要 wss://。純 ws:// 喺登記嗰陣就會被拒,因為我哋串流畀你嘅係真實客戶嘅錄音。你張憑證要通過公開根憑證驗證——自簽、過期、缺中間憑證,或者憑證唔覆蓋嗰個主機名,喺連線測試度全部都會失敗。

我哋唔會send任何 header 或者 token。如果你想要一個共用密鑰,就放喺你登記嗰條網址入面(路徑或者查詢字串),連線嗰陣自己驗。

一次連線一段語音。唔好喺連線之間保留狀態;我哋可能同時開幾條,上限係你自己申報嘅併發數。

整個交換過程,按次序

語音同推理兩個支柱都係咁行——語音入、文字返。另外兩個支柱唔同,喺下面。

  1. 1

    我哋開連線,然後send hello我哋send

    一駁通就send。入面講明係邊個支柱、段語音幾長同乜格式,以及你要遵守嘅限時。請直接讀 deadline_ms,唔好自己計——某一輪可以放寬佢。

    {"type":"hello","protocol":"1.2","run_item_id":"3f2a…","pillar":"asr","clip":{"sample_rate":16000,"encoding":"pcm_s16le_16k","duration_ms":18000},"deadline_ms":9000,"formats":["pcm_s16le_16k"]}
  2. 2

    你回覆 ready你send

    呢一步唔係可選,而且係最多人漏咗嗰步。收到呢個訊框之前我哋唔會串流任何嘢;駁咗上嚟但係唔出聲嘅端點,五秒之後會判為握手未完成。accept 嗰個值唔會拎去議價——我哋兩邊都只係send一種格式。

    {"type":"ready","accept":"pcm_s16le_16k"}
  3. 3

    我哋以二進位訊框串流語音我哋send

    係真正嘅 WebSocket 二進位訊框,唔係 base64,亦唔係包喺 JSON 入面。每個 3200 bytes——即係 100 毫秒嘅 16kHz 單聲道 16-bit PCM——而喺正式測試入面佢哋會按真人講嘢嘅速度,每 100 毫秒到一個。WAV 標頭喺send之前已經剝走:唔好搵 RIFF,冇嘅。

  4. 4

    我哋send end_of_utterance我哋send

    即係來電者講完嘢喇。你嘅限時由呢個訊框開始計——唔係由 hello,亦唔係由第一個語音位元組。

    {"type":"end_of_utterance"}
  5. 5

    你send轉寫結果你send

    第一個就決定咗呢段語音;之後嘅會被忽略,因為我哋已經收線。用繁體字、香港用法、口語就照口語寫。兩邊喺計分前都會經過已公開嘅正規化規格,所以唔使自己預先正規化。

    {"type":"transcript","text":"你好,我想claim返個醫療費"}

一個本身已經行得嘅伺服器

呢兩個原封不動就跑得起,用個假模型都過到連線測試。你淨係要換一個函數。

server.py
# A Cantonese Voice Benchmark endpoint, in about forty lines.
# python -m pip install websockets
# python server.py      →  ws://0.0.0.0:8080
#
# Put your recogniser in transcribe(). Everything else is the protocol.
import asyncio, json, os
from websockets.asyncio.server import serve

CHUNK = 3200  # bytes per frame we receive: 100ms of 16kHz mono s16le


def transcribe(pcm: bytes) -> str:
    """Raw 16kHz mono signed 16-bit PCM. No WAV header — we strip it."""
    return "你好"  # ← your model here


async def handle(ws):
    audio = bytearray()
    async for message in ws:
        # Binary frames are the caller's voice, arriving while they speak.
        if isinstance(message, (bytes, bytearray)):
            audio.extend(message)
            continue

        frame = json.loads(message)

        if frame["type"] == "hello":
            # Nothing is streamed until you answer this. An endpoint that
            # stays silent here is failed as an incomplete handshake.
            await ws.send(json.dumps({"type": "ready", "accept": "pcm_s16le_16k"}))

        elif frame["type"] == "end_of_utterance":
            # The deadline clock starts at this frame, not at hello.
            text = transcribe(bytes(audio))
            await ws.send(json.dumps({"type": "transcript", "text": text}))
            return  # one clip per connection


async def main():
    async with serve(handle, "0.0.0.0", int(os.environ.get("PORT", 8080))):
        await asyncio.get_running_loop().create_future()

asyncio.run(main())
server.mjs
// A Cantonese Voice Benchmark endpoint, in about forty lines.
// npm i ws  &&  node server.mjs      →  ws://0.0.0.0:8080
//
// Put your recogniser in transcribe(). Everything else is the protocol.
import { WebSocketServer } from 'ws';

const CHUNK = 3200; // bytes per frame we send: 100ms of 16kHz mono s16le

/** Raw 16kHz mono signed 16-bit PCM. No WAV header — we strip it. */
function transcribe(pcm) {
  return '你好'; // ← your model here
}

const wss = new WebSocketServer({ port: Number(process.env.PORT ?? 8080) });

wss.on('connection', (ws) => {
  const audio = [];

  ws.on('message', async (data, isBinary) => {
    // Binary frames are the caller's voice, arriving while they speak.
    if (isBinary) { audio.push(data); return; }

    const frame = JSON.parse(data.toString());

    if (frame.type === 'hello') {
      // Nothing is streamed until you answer this. An endpoint that stays
      // silent here is failed as an incomplete handshake.
      ws.send(JSON.stringify({ type: 'ready', accept: 'pcm_s16le_16k' }));
    }

    if (frame.type === 'end_of_utterance') {
      // The deadline clock starts at this frame, not at hello.
      const text = transcribe(Buffer.concat(audio));
      ws.send(JSON.stringify({ type: 'transcript', text }));
      ws.close();
    }
  });
});

每個支柱有咩唔同

同一套協定,四種用法。最多人中伏嘅係最後一欄——計時由邊度開始。

支柱我哋send乜你回乜限時由邊度計
ASR——語音轉文字hello,然後語音transcriptend_of_utterance
LLM——理解推理hello(連題目),然後語音答案,用 transcript 送返end_of_utterance
TTS——語音合成hello,入面有文字speaking、語音、turn_donehello
E2E——完整對話有編號嘅 turn 訊框speaking、語音、turn_done——同一個編號每次 end_of_utterance

理解推理,係要答,唔係要轉寫

呢個支柱送嘅語音同語音辨識一樣,但問嘅嘢唔同,所以 hello 會帶一個 task 欄位。請答佢——唔好送轉寫返嚟。一個只係重複通話內容嘅回覆會當作冇答,四項評分全部取最低,而喺 task 未送出之前,參賽者遇到嘅正正就係咁。每一題嘅 task 都係同一句:

Answer the caller. In Cantonese, say what they need and what was agreed, grounded only in what the call actually contains. Do not repeat the call back — a transcript is not an answer — and do not invent coverage, amounts or policy terms that were not said.
{"type":"hello","protocol":"1.2","run_item_id":"7b0e…","pillar":"llm","clip":{"sample_rate":16000,"encoding":"pcm_s16le_16k","duration_ms":18000},"task":"Answer the caller. In Cantonese, say what they need and what was agreed, grounded only in what the call actually contains. Do not repeat the call back — a transcript is not an answer — and do not invent coverage, amounts or policy terms that were not said.","deadline_ms":9000,"formats":["pcm_s16le_16k"]}
{"type":"transcript","text":"客戶要求將自動轉帳戶口改為儲蓄戶口,可以喺網上自助辦理。三月保費已於廿號扣除。"}

語音合成,方向係倒轉嘅

我哋喺 hello 入面send文字,唔會send任何語音。你一有第一個位元組就即刻send speaking——嗰個訊框就係你嘅首位元組延遲,因為來電者聽到嘅係第一個音節,唔係最後一個。

tts
# TTS runs the other way: we send text, you send audio back.
# The deadline starts at hello — there is no utterance to end.

if frame["type"] == "hello":
    await ws.send(json.dumps({"type": "ready", "accept": "pcm_s16le_16k"}))

    # Send this the moment the first byte exists. Your latency is measured
    # HERE, not at turn_done — batching the two publishes your slowest number.
    await ws.send(json.dumps({"type": "speaking", "index": 0}))

    pcm = synthesize(frame["text"])          # 16kHz mono s16le, no WAV header
    for i in range(0, len(pcm), 3200):
        await ws.send(pcm[i:i + 3200])   # binary frames

    await ws.send(json.dumps({"type": "turn_done", "index": 0}))

對話,即係同一條連線上面幾個回合

下面兩條規則,決定咗你係被量度緊,定係被記錄為冇出聲。兩樣都真係中過。

e2e
# E2E is a conversation: several turns on one connection, each numbered.
# Two rules decide whether your answers are seen at all.

if frame["type"] == "turn":
    #  1. Do NOT answer here. We are still playing the caller's turn and are
    #     not yet listening — an answer sent now lands before anything reads
    #     it, and the turn is recorded as unanswered.
    pending = frame                          # remember it, say nothing

elif frame["type"] == "end_of_utterance":
    #  2. Answer now, and carry the SAME index back. We match strictly on it;
    #     a reply with the wrong index, or none, is invisible.
    i = pending["index"]
    await ws.send(json.dumps({"type": "speaking", "index": i}))
    for chunk in synthesize(reply_to(pending["text"])):
        await ws.send(chunk)
    await ws.send(json.dumps({"type": "turn_done", "index": i}))

elif frame["type"] == "barge_in":
    # The caller has interrupted. Stop speaking and send turn_done for that
    # index. How fast you yield the floor is the thing being measured.
    stop_speaking()
    await ws.send(json.dumps({"type": "turn_done", "index": frame["index"]}))
{"type":"turn","index":2,"text":"我想問下我份保單保唔保牙科","audio":{"sample_rate":16000,"encoding":"pcm_s16le_16k","duration_ms":400},"expect_interruption":false}
{"type":"barge_in","index":2}

限時

係段語音長度嘅一半,下限 3 秒:max(3000, duration_ms × 0.5)。一段 60 秒嘅語音畀你 30 秒;一段 4 秒嘅仍然畀你 3 秒。個數值一定喺 deadline_ms 入面——請喺嗰度讀,唔好自己計,因為某一輪可以用唔同嘅系數。

係用我哋部鐘計,由我哋嘅 end_of_utterance 去到你第一個回應位元組。我哋自己嘅連線同握手時間會另外量,永遠唔會計入你嗰個數。

過咗限時之後我哋仲會再聽一秒半——唔係為咗計分,而係為咗可以話返畀你聽「遲咗 412 毫秒」而唔係「冇回應」。無論點樣呢一題都係失敗;呢段寬限期改變嘅係我哋講得出乜嘢,唔係你攞到幾多分。

用聲音回答嘅範疇——語音合成同對話——你第一段聲音到咗就當趕到限時。之後「講完」另有時限:我哋要求嗰句嘅長度嘅 3 倍,最少 15 秒。你個系統係邊講邊傳都冇問題:六秒嘅廣東話本來就唔可能喺三秒限時入面傳完,亦都唔需要。講完一句就傳 turn_done;如果時限先用完,我哋就計嗰陣為止收到嘅聲音,並且會明講。

會出咩事,同埋我哋會點同你講

下面每一項都會喺連線測試度出現,連埋出事嗰段語音一齊顯示。凡係我哋嘅責任,都唔會用掉你嘅配額。

發生咗乜你會見到
你張憑證驗唔到TLS 喺你個端點回應之前就失敗咗,連 OpenSSL 嘅代碼一齊顯示。自簽、過期、缺中間憑證、主機名唔啱,全部歸呢一類。
駁咗上嚟但係冇send ready你個端點接受咗連線之後乜都冇send——連我哋等緊嗰個訊框嘅原文一齊顯示。
DNS 解析唔到你登記嗰個端點嘅 DNS 解析唔到。
連線被拒駁唔到上去。請檢查有冇將我哋嘅出口位址加入白名單。
限時之前乜都冇到喺限時之內冇回應。當作完全答唔中計分。
遲咗先到會顯示遲咗幾多毫秒。唔計入延遲統計,但當作一次失敗。
一個我哋解析唔到嘅訊框回應唔符合協定格式。喺連線上面印一行日誌就足以造成呢個結果——日誌請寫去 stderr,千祈唔好寫入連線。
你send咗 error 訊框會顯示為你個端點回傳 502;除非代碼係 UNSUPPORTED_FORMAT,嗰個會報告為格式協商失敗。
空白嘅轉寫當作完全答唔中計分並標記覆核——同「失敗」分開處理,因為一個乜都唔回嘅系統,本身就係關於嗰個系統嘅一個真實結果。
我哋嘅語音儲存出事係我哋嘅錯。呢一題會重跑,唔會用掉你嘅配額。

四個最多人中伏嘅位

語音係冇標頭嘅
我哋喺串流之前剝走咗 44 bytes 嘅 WAV 標頭,所以到你嗰度嘅係純 PCM:16kHz、單聲道、16-bit 有號、小端序。當佢係 WAV 咁開會乜都開唔到。
一切都係由 ready 開始
唔係 hello。一個收到 hello 就開工、但係唔回覆嘅伺服器,一啲語音都收唔到,而且以前仲會令成次測試吊死,同時佔住你一個併發位。
E2E 要喺 end_of_utterance 之後先答
唔係喺 turn 嗰陣。來電者嘅回合仲播緊嗰陣我哋未開始聽你嘅回覆,所以喺 turn 嗰陣send嘅答案,會早過任何人讀佢大約一秒到達,嗰個回合就會被記錄為冇答。
保持條連線乾淨
任何唔係我哋認得嘅訊框,都會即刻令嗰段語音失敗——一行印咗落連線嘅除錯訊息就夠。

證明佢行得

十秒、三段真實語音,任何失敗都有一句睇得明嘅診斷。想跑幾多次都得;唔會用掉配額。

  1. 1

    先指向我哋嗰個

    你嘅參賽者頁面提供一個參考端點,佢正確咁實作咗上面全部嘢。四個支柱都登記佢再跑測試——即係喺你未寫過一行code之前,就已經證明咗平台嗰邊冇問題。

  2. 2

    登記你自己嘅

    參賽者頁面 → 端點。每個支柱一條網址,加埋你同一時間食得起幾多條連線。我哋唔會超過。

  3. 3

    跑連線測試

    三段最短嘅樣本語音,用你申報嘅併發數,同正式測試一樣嘅節奏同逾時設定。三段都要過。

  4. 4

    如果唔過,我哋會測埋自己

    自動用同樣嗰三段語音跑一次。所以個結果會直接話你知係你嗰邊定我哋嗰邊出事,唔使你自己估。

  5. 5

    然後就可以練習

    練習語料包喺你嘅參賽者頁面度下載:同樣嘅錄音連參考轉寫,等你可以喺約正式測試時段之前,自己計返個錯誤率。

如果卡住咗

電郵 [email protected],講低係邊個支柱,如果有嘅話連連線測試嗰個編號一齊寄嚟。我哋一個工作天內回覆。一個你睇唔明嘅錯誤訊息,係我哋寫得唔好,唔係你應該要識嘅嘢。