Building the endpoint.

The shape of it

The exchange, in order

Speech and reasoning both work this way — audio in, text back. The other two pillars differ and are below.

1. 1

   We open the socket and send hellowe send

   Immediately on connect. It names the pillar, the clip’s length and format, and the deadline you will be held to. Read deadline_ms rather than computing it — a round can relax it.

   {"type":"hello","protocol":"1.2","run_item_id":"3f2a…","pillar":"asr","clip":{"sample_rate":16000,"encoding":"pcm_s16le_16k","duration_ms":18000},"deadline_ms":9000,"formats":["pcm_s16le_16k"]}

2. 2

   You answer readyyou send

   This is not optional and it is the step people miss. Nothing is streamed until this frame arrives; an endpoint that connects and stays silent is failed as an incomplete handshake after five seconds. The accept value is not negotiated — we send one format either way.

   {"type":"ready","accept":"pcm_s16le_16k"}

3. 3

   We stream the audio as binary frameswe send

   Real WebSocket binary frames, not base64 and not wrapped in JSON. 3200 bytes each — 100ms of 16kHz mono signed 16-bit PCM — and during a scored run they arrive paced, one every 100ms, the way a person actually speaks. The WAV header is stripped before we send: do not look for RIFF, there is none.

4. 4

   We send end_of_utterancewe send

   The caller has stopped talking. Your deadline is measured from this frame — not from hello, and not from the first byte of audio.

   {"type":"end_of_utterance"}

5. 5

   You send the transcriptyou send

   The first one settles the clip; anything after it is ignored because we close. Traditional characters, Hong Kong conventions, spoken Cantonese as spoken. Both sides pass through the published normalization before scoring, so do not pre-normalise.

   {"type":"transcript","text":"你好，我想claim返個醫療費"}

A server that already works

Both of these run as they stand and pass the connection test with a stub model. Replace one function.

# A Cantonese Voice Benchmark endpoint, in about forty lines.
# python -m pip install websockets
# python server.py      →  ws://0.0.0.0:8080
#
# Put your recogniser in transcribe(). Everything else is the protocol.
import asyncio, json, os
from websockets.asyncio.server import serve

CHUNK = 3200  # bytes per frame we receive: 100ms of 16kHz mono s16le


def transcribe(pcm: bytes) -> str:
    """Raw 16kHz mono signed 16-bit PCM. No WAV header — we strip it."""
    return "你好"  # ← your model here


async def handle(ws):
    audio = bytearray()
    async for message in ws:
        # Binary frames are the caller's voice, arriving while they speak.
        if isinstance(message, (bytes, bytearray)):
            audio.extend(message)
            continue

        frame = json.loads(message)

        if frame["type"] == "hello":
            # Nothing is streamed until you answer this. An endpoint that
            # stays silent here is failed as an incomplete handshake.
            await ws.send(json.dumps({"type": "ready", "accept": "pcm_s16le_16k"}))

        elif frame["type"] == "end_of_utterance":
            # The deadline clock starts at this frame, not at hello.
            text = transcribe(bytes(audio))
            await ws.send(json.dumps({"type": "transcript", "text": text}))
            return  # one clip per connection


async def main():
    async with serve(handle, "0.0.0.0", int(os.environ.get("PORT", 8080))):
        await asyncio.get_running_loop().create_future()

asyncio.run(main())

// A Cantonese Voice Benchmark endpoint, in about forty lines.
// npm i ws  &&  node server.mjs      →  ws://0.0.0.0:8080
//
// Put your recogniser in transcribe(). Everything else is the protocol.
import { WebSocketServer } from 'ws';

const CHUNK = 3200; // bytes per frame we send: 100ms of 16kHz mono s16le

/** Raw 16kHz mono signed 16-bit PCM. No WAV header — we strip it. */
function transcribe(pcm) {
  return '你好'; // ← your model here
}

const wss = new WebSocketServer({ port: Number(process.env.PORT ?? 8080) });

wss.on('connection', (ws) => {
  const audio = [];

  ws.on('message', async (data, isBinary) => {
    // Binary frames are the caller's voice, arriving while they speak.
    if (isBinary) { audio.push(data); return; }

    const frame = JSON.parse(data.toString());

    if (frame.type === 'hello') {
      // Nothing is streamed until you answer this. An endpoint that stays
      // silent here is failed as an incomplete handshake.
      ws.send(JSON.stringify({ type: 'ready', accept: 'pcm_s16le_16k' }));
    }

    if (frame.type === 'end_of_utterance') {
      // The deadline clock starts at this frame, not at hello.
      const text = transcribe(Buffer.concat(audio));
      ws.send(JSON.stringify({ type: 'transcript', text }));
      ws.close();
    }
  });
});

What changes per pillar

One protocol, four uses of it. The column that catches people is the last one — where the clock starts.

Reasoning, which is answering rather than transcribing

This pillar sends the same audio as recognition and asks a different question, so hello carries a task field. Answer it — do not send back a transcript. A reply that repeats the call is scored as a non-answer and takes the floor on every dimension, which is exactly what happened to entrants before the task was sent. The task is the same on every item:

Answer the caller. In Cantonese, say what they need and what was agreed, grounded only in what the call actually contains. Do not repeat the call back — a transcript is not an answer — and do not invent coverage, amounts or policy terms that were not said.

{"type":"hello","protocol":"1.2","run_item_id":"7b0e…","pillar":"llm","clip":{"sample_rate":16000,"encoding":"pcm_s16le_16k","duration_ms":18000},"task":"Answer the caller. In Cantonese, say what they need and what was agreed, grounded only in what the call actually contains. Do not repeat the call back — a transcript is not an answer — and do not invent coverage, amounts or policy terms that were not said.","deadline_ms":9000,"formats":["pcm_s16le_16k"]}

{"type":"transcript","text":"客戶要求將自動轉帳戶口改為儲蓄戶口，可以喺網上自助辦理。三月保費已於廿號扣除。"}

Speech, which runs backwards

We send text in hello and no audio at all. Send speaking the moment your first byte exists — that frame is your time-to-first-byte, and a caller hears the first syllable, not the last.

# TTS runs the other way: we send text, you send audio back.
# The deadline starts at hello — there is no utterance to end.

if frame["type"] == "hello":
    await ws.send(json.dumps({"type": "ready", "accept": "pcm_s16le_16k"}))

    # Send this the moment the first byte exists. Your latency is measured
    # HERE, not at turn_done — batching the two publishes your slowest number.
    await ws.send(json.dumps({"type": "speaking", "index": 0}))

    pcm = synthesize(frame["text"])          # 16kHz mono s16le, no WAV header
    for i in range(0, len(pcm), 3200):
        await ws.send(pcm[i:i + 3200])   # binary frames

    await ws.send(json.dumps({"type": "turn_done", "index": 0}))

Conversation, which is several turns on one socket

The two rules below are the difference between being measured and being recorded as silent. Both have caught real entrants.

# E2E is a conversation: several turns on one connection, each numbered.
# Two rules decide whether your answers are seen at all.

if frame["type"] == "turn":
    #  1. Do NOT answer here. We are still playing the caller's turn and are
    #     not yet listening — an answer sent now lands before anything reads
    #     it, and the turn is recorded as unanswered.
    pending = frame                          # remember it, say nothing

elif frame["type"] == "end_of_utterance":
    #  2. Answer now, and carry the SAME index back. We match strictly on it;
    #     a reply with the wrong index, or none, is invisible.
    i = pending["index"]
    await ws.send(json.dumps({"type": "speaking", "index": i}))
    for chunk in synthesize(reply_to(pending["text"])):
        await ws.send(chunk)
    await ws.send(json.dumps({"type": "turn_done", "index": i}))

elif frame["type"] == "barge_in":
    # The caller has interrupted. Stop speaking and send turn_done for that
    # index. How fast you yield the floor is the thing being measured.
    stop_speaking()
    await ws.send(json.dumps({"type": "turn_done", "index": frame["index"]}))

{"type":"turn","index":2,"text":"我想問下我份保單保唔保牙科","audio":{"sample_rate":16000,"encoding":"pcm_s16le_16k","duration_ms":400},"expect_interruption":false}

{"type":"barge_in","index":2}

The deadline

What goes wrong, and what we will tell you

Every one of these appears in the connection test with the sentence below, against the clip that produced it. Nothing that is our fault costs you quota.

Four things that catch people

The audio has no header

We strip the 44-byte WAV header before streaming, so what arrives is raw PCM: 16kHz, mono, signed 16-bit, little-endian. Code that opens it as a WAV finds nothing to open.

ready is what starts everything

Not hello. A server that begins work on hello and never answers gets no audio at all, and used to hang the run holding one of your concurrency slots.

On E2E, answer after end_of_utterance

Not on turn. While the caller’s turn is playing we are not yet listening for your reply, so an answer sent on turn arrives about a second before anything reads it and the turn is recorded as unanswered.

Keep the socket clean

Anything that is not a frame we recognise fails the clip immediately — one debug line printed to the connection is enough.

Proving it works

Ten seconds, three real clips, and a readable diagnosis for anything that fails. Run it as often as you like; it costs no quota.

1. 1

   Point at ours first

   Your portal offers a reference endpoint that implements all of this correctly. Register it for all four pillars and run the test — that proves the platform side works before you have written anything.

2. 2

   Register your own

   Portal → Endpoints. One URL per pillar, plus the number of connections you can take at once. We never exceed it.

3. 3

   Run the connection test

   Three of the shortest sample clips, at your declared concurrency, with the same pacing and timeouts a scored run uses. All three must pass.

4. 4

   If it fails, we test ours

   Automatically, on the same three clips. So the result tells you whether the fault is yours or ours, rather than leaving you to guess.

5. 5

   Then practise

   The sample pack is a download in your portal: the same recordings with their reference transcripts, so you can compute your own error rate before booking a scored window.

If you are stuck