Questions

Straight answers.

Search the answers

Entering

Do I need a full voice stack?: For a rank, yes. The Core Score is recognition, understanding and speech — ASR + LLM + TTS — rescaled to 100, and it is withheld until all three are final, so an entry missing one is published as incomplete and sits outside the ranking rather than scoring an average over a third of the work. Conversation is optional and sits in the Full Score. You may still enter fewer pillars: each one is published on its own, and a strong recogniser is never hidden behind a missing synthesiser. Which pillars a round ranks on is stated on each pillar page while the round is open.
I have an account. What stands between me and a scored run?: One step: send your application from the portal, naming the pillars you are entering and what you have built. It is approved the moment you send it — the organisers issuing your account was the vetting — and scored runs open in those pillars whenever a round is open. Practice works before that.
Do I have to write a system description?: Yes, before final scoring — one to two pages, written in the portal against six questions. Your Core Score is unaffected; the Full Score is withheld until it is in, and the leaderboard says so rather than leaving you guessing.

Integrating

For recognition, do I label speakers, and what about overlapping speech and lazy pronunciation?: No labels. The transcript frame has one text field and no speaker field, and the answer key holds both parties’ words with no labels in it — so do not add “客戶：” or “Agent:”: normalisation removes the colon, not the word, and the label is scored as extra characters. The audio is one mono channel with both sides of the call mixed; transcribe everything both people say, in the order it was said, including the words spoken over each other — a missed overlap counts as deleted characters, and the run page’s “right words, wrong place” shows what ordering cost you. Write words, not pronunciations: Traditional characters, Hong Kong usage, spoken Cantonese as spoken (佢, 嘅, 咗), and a lazy pronunciation does not change the character — 你 said “lei” is still 你. Fillers such as 呃 and 嗯 are speech and are scored.
Does the understanding pillar need the insurer’s SOP or a knowledge base?: None is issued, and no internal rule of the insurer’s is ever required to answer. How you build your system is yours: a retrieval index or a fine-tune over public insurance material — product pages, published procedures, ordinary Hong Kong practice — is allowed and usually helps with terminology and register. The one training restriction is the evaluation audio itself, which may not be trained on, redistributed or published. What no outside source can supply is the call-specific facts: this caller’s balance, this policy’s status, what was agreed on this call. Those come from the recording, and no policy data is supplied or queryable — if a fact is not in the call, nobody scoring it has it either. The judge is given the same task you are, that call’s verified transcript, and the key facts an organiser recorded from it; policy accuracy asks whether a citation matched what the call establishes, not whether you knew an internal rule. Where the call does not settle something, the answer that scores says what the call supports, names what it cannot confirm, and offers the next step.
How long does integration take?: Two to three hours for ASR, about a day for the full stack. You implement a WebSocket server and we connect to it. Once the URL is registered, the portal runs a connection test against three real clips and gives you a readable diagnosis of anything that fails, so you find out from us rather than from a scored run.
Why do you call my endpoint instead of taking a file?: Because latency and turn-taking cannot be measured from a file, and those are 25% of the score. A batch upload path exists as a fallback for recognition only, labelled on the leaderboard. It is not a Core Score on its own: Core also needs understanding and speech, which are measured only over a connection.
What happens if my endpoint goes down mid-window?: The run reports insufficient data rather than a bad score, and it does not consume your quota. Failures we caused are attributed to us and re-run for free. A downed endpoint is never published as a 100% error rate, because that would be a false statement about your system.
Can I see which clips I got wrong?: On practice runs, yes, with the reference transcript alongside your output. On scored runs you see per-item status and your own response, but not the ground truth.

How results work

Does a scored run count automatically?: No. A scored run is a draft until you press Submit on its own page. Until then nobody but you can see it, and it is not on the board and not in the ranking. Nothing submits it for you, and once the round closes it can no longer be submitted. The board shows the latest run you submitted for each pillar, so a run that came out worse costs you nothing if you leave it unsubmitted — and your portal lists any pillar where a scored run is still waiting.
Who judges the LLM pillar?: Two Amazon Nova models — Pro and Lite — score every answer three times each against published anchors, at temperature 0. The median is taken, the spread between runs is kept, and an item where the two models differ by more than one rubric point is marked contested rather than averaged into agreement. Before any model is called, a deterministic check floors a reply that reproduces the call: that is a string question with a certain answer, and on short calls a model gets it wrong. Both models come from one vendor, so this is independence from one model’s quirks and not from one vendor’s; two providers is still the goal and this page will say so until it is true. Choosing them was a measurement, not a preference: across five random production clips a genuine answer ranked above a copy-paste, an echo and a non-answer, and a frontier pair cost twenty times more for the same work. Until 8 September 2026 this pillar was scored by a deterministic word-overlap rubric instead. It was wrong in a way worth stating plainly: measured on a real call, a correct Cantonese answer scored 1.41 of 5 while the first 30% of the call pasted back scored 4.97. Every score it produced has been voided. A deployment without model credentials scores no reasoning runs at all, rather than falling back to that rubric. Validation against human raters happens before this pillar counts either way.
Who rates the speech?: Paid native Cantonese listeners who pass a calibration set first. They never see which system produced a sample, playback order is randomised, and ratings are only accepted once the audio has actually finished playing.
Will I be named?: Only if you choose to be. Codename by default, opt-in to be named, reversible until publication. After publication a named entry can still be un-named by asking the organisers, and the change is recorded.
Why are three teams sharing rank 2?: Because their confidence intervals overlap. A round draws a few dozen clips, and over a draw that size a difference of a fraction of a percent is noise — printing a strict order over noise is false precision. The exact number of clips in the current round is on the leaderboard. Tied means we cannot tell them apart, and saying so is more useful than inventing a winner.

The data

Can I train on the data?: No. You may use it to produce submissions during the edition, and copies are deleted within thirty days of it closing.

Something not answered here? Get in touch.