接入
你架一個 WebSocket 伺服器,我哋做客戶端駁上去。呢一版就係成套通訊協定——兩個方向嘅每一個訊框,加埋一個貼咗就行得嘅伺服器。語音支柱一般兩至三個鐘搞掂。
我哋係客戶端。你唔使打嚟我哋度:你自己架伺服器、登記個網址,測試開始嗰陣我哋會駁上去。咁樣我哋先可以喺我哋揀嘅條件下量你個系統,而唔係喺你揀嘅條件下。
個網址一定要 wss://。純 ws:// 喺登記嗰陣就會被拒,因為我哋串流畀你嘅係真實客戶嘅錄音。你張憑證要通過公開根憑證驗證——自簽、過期、缺中間憑證,或者憑證唔覆蓋嗰個主機名,喺連線測試度全部都會失敗。
我哋唔會send任何 header 或者 token。如果你想要一個共用密鑰,就放喺你登記嗰條網址入面(路徑或者查詢字串),連線嗰陣自己驗。
一次連線一段語音。唔好喺連線之間保留狀態;我哋可能同時開幾條,上限係你自己申報嘅併發數。
語音同推理兩個支柱都係咁行——語音入、文字返。另外兩個支柱唔同,喺下面。
一駁通就send。入面講明係邊個支柱、段語音幾長同乜格式,以及你要遵守嘅限時。請直接讀 deadline_ms,唔好自己計——某一輪可以放寬佢。
{"type":"hello","protocol":"1.2","run_item_id":"3f2a…","pillar":"asr","clip":{"sample_rate":16000,"encoding":"pcm_s16le_16k","duration_ms":18000},"deadline_ms":9000,"formats":["pcm_s16le_16k"]}呢一步唔係可選,而且係最多人漏咗嗰步。收到呢個訊框之前我哋唔會串流任何嘢;駁咗上嚟但係唔出聲嘅端點,五秒之後會判為握手未完成。accept 嗰個值唔會拎去議價——我哋兩邊都只係send一種格式。
{"type":"ready","accept":"pcm_s16le_16k"}係真正嘅 WebSocket 二進位訊框,唔係 base64,亦唔係包喺 JSON 入面。每個 3200 bytes——即係 100 毫秒嘅 16kHz 單聲道 16-bit PCM——而喺正式測試入面佢哋會按真人講嘢嘅速度,每 100 毫秒到一個。WAV 標頭喺send之前已經剝走:唔好搵 RIFF,冇嘅。
即係來電者講完嘢喇。你嘅限時由呢個訊框開始計——唔係由 hello,亦唔係由第一個語音位元組。
{"type":"end_of_utterance"}第一個就決定咗呢段語音;之後嘅會被忽略,因為我哋已經收線。用繁體字、香港用法、口語就照口語寫。兩邊喺計分前都會經過已公開嘅正規化規格,所以唔使自己預先正規化。
{"type":"transcript","text":"你好,我想claim返個醫療費"}呢兩個原封不動就跑得起,用個假模型都過到連線測試。你淨係要換一個函數。
# A Cantonese Voice Benchmark endpoint, in about forty lines.
# python -m pip install websockets
# python server.py → ws://0.0.0.0:8080
#
# Put your recogniser in transcribe(). Everything else is the protocol.
import asyncio, json, os
from websockets.asyncio.server import serve
CHUNK = 3200 # bytes per frame we receive: 100ms of 16kHz mono s16le
def transcribe(pcm: bytes) -> str:
"""Raw 16kHz mono signed 16-bit PCM. No WAV header — we strip it."""
return "你好" # ← your model here
async def handle(ws):
audio = bytearray()
async for message in ws:
# Binary frames are the caller's voice, arriving while they speak.
if isinstance(message, (bytes, bytearray)):
audio.extend(message)
continue
frame = json.loads(message)
if frame["type"] == "hello":
# Nothing is streamed until you answer this. An endpoint that
# stays silent here is failed as an incomplete handshake.
await ws.send(json.dumps({"type": "ready", "accept": "pcm_s16le_16k"}))
elif frame["type"] == "end_of_utterance":
# The deadline clock starts at this frame, not at hello.
text = transcribe(bytes(audio))
await ws.send(json.dumps({"type": "transcript", "text": text}))
return # one clip per connection
async def main():
async with serve(handle, "0.0.0.0", int(os.environ.get("PORT", 8080))):
await asyncio.get_running_loop().create_future()
asyncio.run(main())
// A Cantonese Voice Benchmark endpoint, in about forty lines.
// npm i ws && node server.mjs → ws://0.0.0.0:8080
//
// Put your recogniser in transcribe(). Everything else is the protocol.
import { WebSocketServer } from 'ws';
const CHUNK = 3200; // bytes per frame we send: 100ms of 16kHz mono s16le
/** Raw 16kHz mono signed 16-bit PCM. No WAV header — we strip it. */
function transcribe(pcm) {
return '你好'; // ← your model here
}
const wss = new WebSocketServer({ port: Number(process.env.PORT ?? 8080) });
wss.on('connection', (ws) => {
const audio = [];
ws.on('message', async (data, isBinary) => {
// Binary frames are the caller's voice, arriving while they speak.
if (isBinary) { audio.push(data); return; }
const frame = JSON.parse(data.toString());
if (frame.type === 'hello') {
// Nothing is streamed until you answer this. An endpoint that stays
// silent here is failed as an incomplete handshake.
ws.send(JSON.stringify({ type: 'ready', accept: 'pcm_s16le_16k' }));
}
if (frame.type === 'end_of_utterance') {
// The deadline clock starts at this frame, not at hello.
const text = transcribe(Buffer.concat(audio));
ws.send(JSON.stringify({ type: 'transcript', text }));
ws.close();
}
});
});
同一套協定,四種用法。最多人中伏嘅係最後一欄——計時由邊度開始。
| 支柱 | 我哋send乜 | 你回乜 | 限時由邊度計 |
|---|---|---|---|
| ASR——語音轉文字 | hello,然後語音 | transcript | end_of_utterance |
| LLM——理解推理 | hello(連題目),然後語音 | 答案,用 transcript 送返 | end_of_utterance |
| TTS——語音合成 | hello,入面有文字 | speaking、語音、turn_done | hello |
| E2E——完整對話 | 有編號嘅 turn 訊框 | speaking、語音、turn_done——同一個編號 | 每次 end_of_utterance |
呢個支柱送嘅語音同語音辨識一樣,但問嘅嘢唔同,所以 hello 會帶一個 task 欄位。請答佢——唔好送轉寫返嚟。一個只係重複通話內容嘅回覆會當作冇答,四項評分全部取最低,而喺 task 未送出之前,參賽者遇到嘅正正就係咁。每一題嘅 task 都係同一句:
Answer the caller. In Cantonese, say what they need and what was agreed, grounded only in what the call actually contains. Do not repeat the call back — a transcript is not an answer — and do not invent coverage, amounts or policy terms that were not said.
{"type":"hello","protocol":"1.2","run_item_id":"7b0e…","pillar":"llm","clip":{"sample_rate":16000,"encoding":"pcm_s16le_16k","duration_ms":18000},"task":"Answer the caller. In Cantonese, say what they need and what was agreed, grounded only in what the call actually contains. Do not repeat the call back — a transcript is not an answer — and do not invent coverage, amounts or policy terms that were not said.","deadline_ms":9000,"formats":["pcm_s16le_16k"]}{"type":"transcript","text":"客戶要求將自動轉帳戶口改為儲蓄戶口,可以喺網上自助辦理。三月保費已於廿號扣除。"}我哋喺 hello 入面send文字,唔會send任何語音。你一有第一個位元組就即刻send speaking——嗰個訊框就係你嘅首位元組延遲,因為來電者聽到嘅係第一個音節,唔係最後一個。
# TTS runs the other way: we send text, you send audio back.
# The deadline starts at hello — there is no utterance to end.
if frame["type"] == "hello":
await ws.send(json.dumps({"type": "ready", "accept": "pcm_s16le_16k"}))
# Send this the moment the first byte exists. Your latency is measured
# HERE, not at turn_done — batching the two publishes your slowest number.
await ws.send(json.dumps({"type": "speaking", "index": 0}))
pcm = synthesize(frame["text"]) # 16kHz mono s16le, no WAV header
for i in range(0, len(pcm), 3200):
await ws.send(pcm[i:i + 3200]) # binary frames
await ws.send(json.dumps({"type": "turn_done", "index": 0}))
下面兩條規則,決定咗你係被量度緊,定係被記錄為冇出聲。兩樣都真係中過。
# E2E is a conversation: several turns on one connection, each numbered.
# Two rules decide whether your answers are seen at all.
if frame["type"] == "turn":
# 1. Do NOT answer here. We are still playing the caller's turn and are
# not yet listening — an answer sent now lands before anything reads
# it, and the turn is recorded as unanswered.
pending = frame # remember it, say nothing
elif frame["type"] == "end_of_utterance":
# 2. Answer now, and carry the SAME index back. We match strictly on it;
# a reply with the wrong index, or none, is invisible.
i = pending["index"]
await ws.send(json.dumps({"type": "speaking", "index": i}))
for chunk in synthesize(reply_to(pending["text"])):
await ws.send(chunk)
await ws.send(json.dumps({"type": "turn_done", "index": i}))
elif frame["type"] == "barge_in":
# The caller has interrupted. Stop speaking and send turn_done for that
# index. How fast you yield the floor is the thing being measured.
stop_speaking()
await ws.send(json.dumps({"type": "turn_done", "index": frame["index"]}))
{"type":"turn","index":2,"text":"我想問下我份保單保唔保牙科","audio":{"sample_rate":16000,"encoding":"pcm_s16le_16k","duration_ms":400},"expect_interruption":false}{"type":"barge_in","index":2}係段語音長度嘅一半,下限 3 秒:max(3000, duration_ms × 0.5)。一段 60 秒嘅語音畀你 30 秒;一段 4 秒嘅仍然畀你 3 秒。個數值一定喺 deadline_ms 入面——請喺嗰度讀,唔好自己計,因為某一輪可以用唔同嘅系數。
係用我哋部鐘計,由我哋嘅 end_of_utterance 去到你第一個回應位元組。我哋自己嘅連線同握手時間會另外量,永遠唔會計入你嗰個數。
過咗限時之後我哋仲會再聽一秒半——唔係為咗計分,而係為咗可以話返畀你聽「遲咗 412 毫秒」而唔係「冇回應」。無論點樣呢一題都係失敗;呢段寬限期改變嘅係我哋講得出乜嘢,唔係你攞到幾多分。
用聲音回答嘅範疇——語音合成同對話——你第一段聲音到咗就當趕到限時。之後「講完」另有時限:我哋要求嗰句嘅長度嘅 3 倍,最少 15 秒。你個系統係邊講邊傳都冇問題:六秒嘅廣東話本來就唔可能喺三秒限時入面傳完,亦都唔需要。講完一句就傳 turn_done;如果時限先用完,我哋就計嗰陣為止收到嘅聲音,並且會明講。
下面每一項都會喺連線測試度出現,連埋出事嗰段語音一齊顯示。凡係我哋嘅責任,都唔會用掉你嘅配額。
| 發生咗乜 | 你會見到 |
|---|---|
| 你張憑證驗唔到 | TLS 喺你個端點回應之前就失敗咗,連 OpenSSL 嘅代碼一齊顯示。自簽、過期、缺中間憑證、主機名唔啱,全部歸呢一類。 |
| 駁咗上嚟但係冇send ready | 你個端點接受咗連線之後乜都冇send——連我哋等緊嗰個訊框嘅原文一齊顯示。 |
| DNS 解析唔到 | 你登記嗰個端點嘅 DNS 解析唔到。 |
| 連線被拒 | 駁唔到上去。請檢查有冇將我哋嘅出口位址加入白名單。 |
| 限時之前乜都冇到 | 喺限時之內冇回應。當作完全答唔中計分。 |
| 遲咗先到 | 會顯示遲咗幾多毫秒。唔計入延遲統計,但當作一次失敗。 |
| 一個我哋解析唔到嘅訊框 | 回應唔符合協定格式。喺連線上面印一行日誌就足以造成呢個結果——日誌請寫去 stderr,千祈唔好寫入連線。 |
| 你send咗 error 訊框 | 會顯示為你個端點回傳 502;除非代碼係 UNSUPPORTED_FORMAT,嗰個會報告為格式協商失敗。 |
| 空白嘅轉寫 | 當作完全答唔中計分並標記覆核——同「失敗」分開處理,因為一個乜都唔回嘅系統,本身就係關於嗰個系統嘅一個真實結果。 |
| 我哋嘅語音儲存出事 | 係我哋嘅錯。呢一題會重跑,唔會用掉你嘅配額。 |
十秒、三段真實語音,任何失敗都有一句睇得明嘅診斷。想跑幾多次都得;唔會用掉配額。
你嘅參賽者頁面提供一個參考端點,佢正確咁實作咗上面全部嘢。四個支柱都登記佢再跑測試——即係喺你未寫過一行code之前,就已經證明咗平台嗰邊冇問題。
參賽者頁面 → 端點。每個支柱一條網址,加埋你同一時間食得起幾多條連線。我哋唔會超過。
三段最短嘅樣本語音,用你申報嘅併發數,同正式測試一樣嘅節奏同逾時設定。三段都要過。
自動用同樣嗰三段語音跑一次。所以個結果會直接話你知係你嗰邊定我哋嗰邊出事,唔使你自己估。
練習語料包喺你嘅參賽者頁面度下載:同樣嘅錄音連參考轉寫,等你可以喺約正式測試時段之前,自己計返個錯誤率。
電郵 [email protected],講低係邊個支柱,如果有嘅話連連線測試嗰個編號一齊寄嚟。我哋一個工作天內回覆。一個你睇唔明嘅錯誤訊息,係我哋寫得唔好,唔係你應該要識嘅嘢。