Prep Eval API

Every speech- and language-scoring engine behind one host and one credential: https://eval.flowb.ai. Each engine is a lane with its own path prefix and its own native request format — the gateway adds shared authentication and, for the team-hosted graders, holds their upstream credentials so you never handle them.

Authentication

Send your eval key on every request:

X-API-Key: <your-eval-key>

Authorization: Bearer <key> is accepted in place of X-API-Key. Failure is 401 {"detail":"missing or invalid credentials"}. Only /health, the lane healths and this page are open.

Send X-Sender-Id: <your-product-id> on every lane. It is the only place you name yourself: eval logs it, shares capacity by it, and hands it to each team upstream in that upstream's own terms (the X-Sender-Id header to go-scoring, the sender_id form field to CPA), so you never fill an upstream's identity field yourself. Ours are flow-eval.vr, flow-eval.atlas and flow-eval.bench; a caller that sends none is labelled flow-eval. Give a bench or a sweep its own id so it is never read as learner traffic.

Health

GET /health
→ {"ok": true, "version": "1.11.0", "lanes": ["pronvi","pronen","shadowscore","speaking","english","chinese"],
   "admission": {"queue_wait_s": 90, "providers": {"prep-grade": {"slots": 10, "per_minute": 29, "sent_last_minute": 0, "in_flight": 0,
     "waiting": 0, "served": 0, "rejected": 0, "abandoned": 0}, …}}}

Lane healths: GET /pronvi/health, GET /pronen/health, GET /shadowscore/health, GET /speaking/health (open, no key).

/pronvi — Vietnamese pronunciation

Segmental correctness (wav2vec2 CTC-GOP) + tone correctness (speaker-relative F0 against per-dialect calibrated profiles), combined into a 0–9 band. Audio is WAV (any sample rate; resampled to 16 kHz mono).

POST /pronvi/score — one syllable

Multipart form:

→ {"target":"má","target_tone":"sac","dialect":"north","band":8.5,"verdict":"pass","percent":94,
   "segmental":{"ok":true,"gop":…,"ratio":…,"transcript":"ma","base_match":true},
   "tone":{"target":"sac","pred":"sac","conf":0.91,"ok":true},
   "speaker_median":112.4, "tone_flags":["má"]?}

Verdicts: pass / almost / retry. tone_flags appears only when the tone missed.

POST /pronvi/score_utterance — a whole turn

Same form fields as /score with target = the full text. Returns utterance-level segmental plus per-syllable tone rows (syllables) and tone_flags.

POST /pronvi/baseline — native-tongue F0 gauge

multipart: audio=<wav>
→ {"median":110.2,"floor":86.1,"ceiling":141.0}   (422 if no voiced pitch)

POST /pronvi/detect — dialect detection

Form fields audio, target, optional speaker_median. Ranks calibrated dialects by how well the take's tone fits; accumulate across several productions before trusting it.

/pronen — English pronunciation by sound

How well each sound of an English text was pronounced in a take, and which sound was actually heard. The text is what the learner said (a transcript) or was asked to say (a script); pronen scores the sounds of those words and never judges whether they were the right words. Source repo csaptu/pronen.

POST /pronen/score

Multipart form: audio (WAV, any rate) and text.

→ {"score": 87,                          // mean over every sound, 0-100
   "words": [{"word": "think", "ipa": "θɪŋk", "score": 75, "start": 0.42, "end": 0.71,
              "degree": "incorrect",          // the word's worst sound
              "phonemes": [{"ipa": "θ", "score": 0, "degree": "incorrect", "heard": "s", "start": 0.40, "end": 0.48}, …]}, …],
   "sound_errors": {"θ": 1, "d": 3},        // incorrect sounds, by expected IPA
   "model": "facebook/wav2vec2-lv-60-espeak-cv-ft"}

• Expected sounds come from espeak-ng en-us in running speech ("a" is /ɐ/, not the letter name). A pronunciation that is the same English sound said another accepted way is not an error: a clear /t/ for the flapped /ɾ/, a British /ɒ/ for /ɑː/, a full /ʌ/ for an unstressed /ɐ/. Contrasts English keeps (/ɪ/–/iː/, /θ/–/s/, /ð/–/d/, /v/–/b/, /æ/–/ɛ/) always count.
• score per sound is goodness of pronunciation: the expected sound against the strongest sound the model heard in its stretch of the take, 100 when the expected sound was the strongest. heard is always an English sound.
• degree uses Prep's three tiers on this scale: incorrect when the expected sound is essentially absent (score < 1, another sound fully dominates), warning ("a little off") under 40, else correct; a word takes its worst sound's. Against speechocean762's expert labels (2,497 utterances), pronen's sentence scores correlate 0.52 with the experts' (word 0.34), and incorrect flags words at 0.23 precision / 0.64 recall: treat a single flag as a hint, a repeated one as a pattern. Repo README § Calibration.
• Deterministic. A word with nothing to say (punctuation alone) has no score. 422 when the text cannot be aligned to the take. A take is scored on one of three GPU workers (aiserver1 GPU0/GPU1, aiserver2: ~0.05 s, ~0.3-0.6 s with the upload), on eval's own CPU (seconds) only when every worker is down; scored_on says which. GET /pronen/health shows each worker's load.

/shadowscore — shadowing match

Scores how closely a learner's take matches a fixed model recording (scripted repeat): utterance intonation + rhythm always (Praat F0, language-agnostic), plus per-word pronunciation / rhythm / intonation via CTC forced alignment (English). Audio is passed as fetchable public URLs.

POST /shadowscore/shadowing/{lang}

{
  "model_url":   "https://…/model.wav",    // "" → GOP-only mode (drill pronunciation read)
  "learner_url": "https://…/take.wav",
  "script":      "the text that was read",  // enables per-word scoring (en)
  "linking":     [{…}],                     // authored linking spots — scored pass, never penalised
  "rate":        1.0,                       // tier playback rate while recording (bleed cancellation and prosody reference)
  "ref_clips":   [{"url":"…","start_ms":0}] // clips that PLAYED during the take (bleed cancellation and prosody reference; optional score:false excludes the other speaker from grading)
}
→ {
  "intonation": 82, "intonation_calibration": "natural-pitch-v2", "rhythm": 76,           // 0–100, vs the model
  "pronunciation": 71,                      // omitted when the aligner can't run
  "words": [{"word":"loaf","pron":68,"rhythm":80,"intonation":77,
             "linked":false,"start":0.42,"end":0.71,
             "letters":[{"char":"l","score":88,"start":0.42,"end":0.49}, …,
                        {"char":"f","score":21,"start":0.64,"end":0.71}]}, …],
  "timings_ms": {"fetch": 120, "prepare": 5, "score": 160, "align": 140}, // diagnostics; align only when per-word ran
  "aligned_on": {"model": "cache", "learner": "http://127.0.0.1:13172"}, // diagnostics: where each clip was aligned
  "scored_on": "https://shadowscore1.flowb.ai",  // diagnostics: which lane worker answered ("eval" when eval did)
  "bleed_db": 9.6,                          // present when bleed cancellation ran
  "cleaned_b64": "<16-bit WAV>"             // the voice-only take — store it over the raw take
}

Audio URLs must be publicly fetchable https — the gateway refuses private/internal destinations.

words[].letters (1.2.0) is the same forced-alignment GOP one level down: every letter of the word with its own 0–100 alignment score and its span in the take, so a caller can say which part of a word was weak (a low last letter is a dropped ending) instead of marking the whole word. It is a letter, not a phone: the aligner is a character CTC (MMS_FA, a-z + apostrophe) with no phone inventory, so th is two rows, a silent letter still gets one, and the rows carry no IPA. pron is their mean, unchanged. Present exactly when pron is — omitted on a linked word, whose boundary letters are reduced on purpose.

Intonation allows one semitone of natural variation and smooths pitch jitter. Matching level words receive credit; words with fewer than five voiced frames are excluded from the intonation average. The score still distinguishes flat or reversed expressive contours. Pronunciation and lexical-tone correctness are separate dimensions.

/speaking/grade — the universal speaking scorer

The endpoint to grade a spoken English answer. Prep's go-scoring and eval's own /speaking grade the same answer at once, and one result comes back in Prep's own shape (the question-level result node go-scoring returns), so a caller that reads Prep's grade reads this one unchanged. The rules:

• One transcript. result.script is eval's settled transcript (several recognizers, settled before scoring) whenever /speaking answered, and every index in the node points into it, Prep's evidence included: each of Prep's spans is moved word by word onto the same words, and one on words Prep misheard is dropped rather than pointed at other words.
• FC, LR and GRA take the higher band of the two graders (a tie stays Prep's). Whoever won the band, the rest of the criterion is Prep's frame: indicators with their 1-5 scores, linking devices, topic words, collocations, complex structures. Its grammar_errors / vocabulary_errors are both graders' errors, one from /speaking that overlaps one of Prep's left out, in script order.
• Words per minute is measured. Prep sends words_per_minute: 0; whenever /speaking answered, fluency_and_coherence.words_per_minute and indicators.rate_of_speech.words_per_minute carry its rate (spoken words over speech time, from the audio's word timings). rate_of_speech's 1-5 score stays Prep's.
• Pronunciation is Prep's band while Prep heard what was said. When Prep's transcript differs from eval's by more than 0.15 (normalized word error rate), Prep judged sounds the learner never made: the band is eval's, picked.pronunciation is "eval" and pronunciation_source says why. The word tree is Prep's (syllables, stress) on every word both graders heard and eval's on the rest, whatever the rate; mispronounced_words and individual_sound_errors count that tree.
• The overall is IELTS's rule over the four bands that result.
• When one grader fails or runs out of time (Prep 60 s, ours 80 s), the other's result is the answer, alone, the moment it has one. Both failing is 502, or 503 + Retry-After when both were at capacity.
• Progressive (per-turn, live) scoring is not here. It stays on Prep (/english), which answers in ~4 s.

POST /speaking/grade

{ "format": "ielts", "part": 1, "question": "…", "audio_url": "https://…/take.wav",
  "language": "en", "display_lang": "vi",
  "group": "session-123" }      // optional: one learner's answers that share it, sent within 0.5 s of each
                                 // other, go to Prep as ONE grade-test (≤ 8), so a session spends one
                                 // of Prep's 30 requests a minute and Prep's verdict speaks about it alone
→ {
  "format": "ielts", "part": 1,
  "result": {                    // Prep's question-level result node, whoever graded it
    "score": 7.0, "script": "…", "proficiency": {"level": 4, "name": "Very Good"},
    "fluency_and_coherence": {…}, "lexical_resource": {…}, "grammatical_range_and_accuracy": {…},
    "pronunciation": {…},
    "grader": "universal",
    "picked": {"pronunciation": "prep", "fluency_and_coherence": "eval", "lexical_resource": "eval",
               "grammatical_range_and_accuracy": "prep"},
    "pronunciation_source": {"reason": "transcript_mismatch", "wer": 0.24},  // only when P is eval's
    "relevance": {"on_topic": true, "note": "…"}, "reasons": {…}   // from /speaking, when it answered
  },
  "graders": {"prep": {"status": "ok", "bands": {…}, "ms": 4120, "transcript": "…"},   // Prep's own hearing
              "eval": {"status": "ok", "bands": {…}, "ms": 25432, "transcript": {…}, "engine": {…}}},
  "test": {"general_comment": {…}, "action_plan": {…}, …},   // Prep's verdict over the grade-test, or null
  "test_answers": 3
}

• Indices. Every start_index/end_index (end exclusive), topic_specific_words[].start/end and pronunciation word start_original_index/end_original_index (end inclusive, Prep's convention) index result.script. No criterion carries a script of its own. An item's original_text is Prep's wording; the words it marks are the script's (they differ at most by fillers, case and punctuation).
• Taxonomy. Every error category, id and en/vi/fr/th name, operation (U/R/M), severity and deducted score, structure category, word degree and word error tag is Prep's value, whichever grader found it.
• When Prep failed the node is /speaking's alone: no linking_devices or topic_specific_words, no stress, and indicators with Prep's raw counts and rates but a 1-5 score only for rate_of_speech.
• Synchronous: ~7-12 s per answer (the slower grader decides; bench 2026-10-05). Give it a client timeout of at least 100 s.

/speaking — exam speaking scorer

Our own scorer alone, for one spoken answer, on the scale of the exam you name; /speaking/grade runs it beside Prep and is the endpoint to grade with. Unlike /english, the transcript is ours and every criterion is ours: the take is transcribed by several recognizers and settled into one transcript before anything is scored, pronunciation is measured from the audio, and whether the answer fits the question is reported apart and never lowers a language band. Synchronous: one request, one result, in about 7 s for a 40–50 s answer (p50 6.7 s, p90 10 s; bench 2026-10-05, CHANGELOG 1.12.1).

POST /speaking/score

{
  "format": "ielts",            // required; the exam whose scale you want — today: ielts
  "part": 1,                    // required; the format's own part id (IELTS 1-3)
  "question": "What do you like to do after school or at the weekend?",
  "audio_url": "https://…/take.wav",   // public https, ≤ 20 MiB, PCM16 WAV
  "language": "en",             // the language spoken — today: en
  "display_lang": "vi"          // language of every explanation: vi en zh (Simplified) zh-TW (Taiwan) th id ja ko
}

Every field is required. A format, part, language or display language not listed is refused 400 with what is supported. TOEIC, TOEFL iBT, PTE and CEFR are planned; each is added only once its rubric is written in and it has been benched against examiner scores.

→ {
  "format": "ielts", "part": 1,
  "transcript": {
    "text": "So, um after school I play some sports. And the sport of my choice is badminton. …",
    "chosen": "consensus",       // or the one engine whose text it is, when nothing was disputed
    "candidates": [{"engine": "parakeet", "text": "… is Padminton. …"},
                   {"engine": "whisper", "text": "… is badminton. …"}],
    "agreement": 0.879,          // share of words the recognizers heard alike
    "disputes": 5                // stretches the recognizers disagreed on, settled by choice
  },
  "scores": {                    // the format's own scale and criterion names
    "overall": 6.5,
    "fluency_coherence": 6.0, "lexical_resource": 6.0,
    "grammatical_range_accuracy": 6.0, "pronunciation": 7.0
  },
  "reasons": {"fluency_coherence": "…", …},          // one sentence each, in display_lang, citing evidence
  "relevance": {"on_topic": true, "note": "…"},     // reported, never folded into a band
  "evidence": {                  // start_index/end_index index transcript.text, end exclusive
    "grammar_errors":    [{"original_text": "…", "corrected_text": "…", "category": "…", "explanation": "…", "start_index": 0, "end_index": 0}],
    "vocabulary_errors": [ … same shape … ],
    "collocations":      [{"original_text": "…", "explanation": "…", "start_index": 0, "end_index": 0}],
    "complex_structures":[{"original_text": "…", "category": "…", "explanation": "…", "start_index": 0, "end_index": 0}],
    "repetitions":       [{"original_text": "the the", "start_index": 206, "end_index": 213}],
    "fillers":           [{"original_text": "um", "start_index": 4, "end_index": 6}],
    "pronunciation": {                       // from /pronen, on the settled transcript
      "words": [{"original_text": "school", "start_index": 0, "end_index": 0, "ipa": "skuːl", "score": 42,
                 "start": 4.36, "end": 5.38, "degree": "warning", "phonemes": [{"ipa": "uː", "heard": "ʊ", "score": 6, "degree": "warning"}, …]}, …],
      "flagged_words": [{"original_text": "…", "start_index": 0, "end_index": 0, "ipa": "…", "score": 0, "degree": "incorrect",
                         "start": 0.0, "end": 0.0, "sounds": [{"ipa": "ð", "heard": "d", "score": 0, "degree": "incorrect"}]}],   // words that are not "correct", with only their sounds that are not
      "sound_errors": {"d": 12, "v": 8, "ð": 7},
      "phoneme_model": "facebook/wav2vec2-lv-60-espeak-cv-ft",
      "mean_score": 67.1, "calibration": "pronen-content-two-anchor-261005"   // the band: mean content-word score
    },
    "words_per_minute": 125, "words": 75, "speech_seconds": 36.1,
    "pauses": 9, "long_pauses": 2, "pause_seconds": 8.5, "mean_run_words": 7.5
  },
  "engine": {"version": "speaking-1", "lane": "fast", "stt": ["parakeet", "whisper"],
             "llm": {"settle": {"tier": "…", "ms": 884}, "score": {"tier": "…", "ms": 5464}},
             "evidence_dropped": 0, "ms": {"fetch": 0, "stt": 0, "settle": 0, "align": 0, "score": 0, "total": 0}}
}

How it scores

1. Transcript first. The take is transcribed by Parakeet (verbatim: it keeps "um" and "the the") and whisper-large-v3-turbo (spells words well, cleans speech away), both on our GPUs (aiserver1 and aiserver2, Cloudflare's whisper and the mini's Parakeet as the last rungs), each pinned. Where they agree the word stands. Where only the verbatim engine heard a filler or a repeat, it is kept: fluency is scored on it. Any other disagreement goes to the fast model, which may only choose between the recognizers' own words, never write its own, so the learner's own grammar and word errors survive.
2. One alignment of the take, by sound, on a GPU (/pronen, ~0.3-0.6 s). It gives which sounds were mispronounced (each sound's score, the sound heard, Prep's degrees, the words that are not correct), and from the same pass the timings: speech time, rate, pauses (≥ 0.5 s between words; long ≥ 1 s) and runs. The pronunciation band is the mean pronen score of the content words (function words' weak forms are connected speech, not error), mapped by a provisional calibration with two measured anchors (native speech 97 → 9, audio aligned to words never said 24 → 1) until examiner-scored takes fit it. Against speechocean762's human experts pronen's sentence scores correlate 0.52, against 0.42 for the letter measure it replaced (1.11.0).
3. The other criteria on evidence. A model rates them against the format's public band descriptors, quoted in full, listing the evidence first. Hesitations, repeats and self-corrections count as fluency evidence, never as grammar or word errors. Evidence is located in the transcript by code; an item that is not there verbatim is dropped (engine.evidence_dropped), so text[start_index:end_index] == original_text always holds.
4. The overall by the format's rule, in code. IELTS: the mean of the four whole criterion bands, to the nearest half band, with a quarter rounding up.

Several answers make a test: score each answer in its own call, then aggregate by the format's rule. For IELTS, take the mean of each criterion, round it to the half band, then the overall of those four.

A take with fewer than three words of speech is 422 "no rateable speech", with the transcript. A recognizer or the model failing is 502, and the detail names which one.

/english — IELTS speaking & writing grading

A transparent proxy to the team's go-scoring-service: same paths, same JSON, under the /english prefix. Speaking answers are fetchable WAV URLs; writing answers are text. The gateway holds the grader's credential — send only your eval key.

One pair of endpoints for everything. Grading is asynchronous: enqueue a session with grade-test, then poll result/{session_id} until it is terminal. That holds for ONE answer as much as for a whole test — one answer is a session with one response. The legacy synchronous grade-question was retired on 2026-09-29 (its service dropped every request past three in flight); calling it now answers 404.

Every body is wrapped in a {"meta":…,"data":…} envelope, success and error alike:

{"meta":{"code":"200000","type":"SUCCESS","message":"Success",
         "service_id":"go-scoring-service","request_id":"cbc2cf50…"},
 "data":{ …the payload… }}

meta.code is a six-digit business code, not the HTTP status (200000 success, 400000 bad request, 401000 unauthenticated, 429000 rate limited). meta.request_id is the team's log-trace handle — quote it when you report a bad grading.

POST /english/api/v1/ielts/speaking/grade-test
{"meta":{"submission_skill_id":"<24 hex>"},
 "responses":[{"question":{"id":1,"title":"…","part":1},"answer":"https://…/answer.wav"}]}
→ {"meta":{…},"data":{"session_id":"…"}}

GET /english/api/v1/ielts/speaking/result/{session_id}
→ {"meta":{…},
   "data":{"session_id":"…","service":"ielts_speaking_test","status":3,
           "status_label":"Completed","progress":100,"score":6.5,
           "result":{"score":6.5,"pronunciation":{…},"fluency_and_coherence":{…},
                     "lexical_resource":{…},"grammatical_range_and_accuracy":{…},
                     "parts":[{"part":1,"questions":[{"question":{…},"answer":"…",
                                                      "result":{ …one answer's grade… }}]}]}}}

Status: 1 pending · 2 in progress · 3 completed · 4 failed, echoed as status_label with a 0–100 progress. data.result is null until the session completes. Poll every 1 s — polls are free (see Capacity). Measured 2026-09-29, enqueue to completed: one spoken answer 3.7 s p50 (10 in flight), one Task 2 essay ~7 s.

One answer's grade is data.result.parts[0].questions[0].result: the four criterion scores, each criterion's indicators tree, and the per-word / per-phoneme pronunciation detail — exactly the node the retired grade-question returned. Writing's criterion keys are task_response (or task_achievement), cohesion_coherence, lexical_resource, grammatical_range_accuracy. submission_skill_id is required and echoed back; any 24-hex id of yours will do.

Writing's grade-test also accepts user_id, meta.submission_id, per-question sample_essay and break_points (the task's requirement points, which drive the addressing_all_parts indicator), a configs block (check_plagiarism, check_ai_content, upgrade_answer, get_tips) and an optional callback_url.

/chinese — Chinese pronunciation (CPA)

A transparent proxy to the team's Chinese Pronunciation Assessment. The upstream credential is injected by the gateway — send only your eval key. Synchronous multipart; send the clip itself as audio_file (preferred), or a fetchable audio_url. Name yourself with the X-Sender-Id header: eval writes CPA's sender_id field from it and replaces any sender_id you send.

POST /chinese/api/v1/scripted_speech_analysis

→ {"has_speech":true,
   "scores":{"pronunciation":87.2,"tone":78.5,"overall":83.9},
   "utterances":[{"words":[{"text":"你","pinyin":"nǐ","syllables":[…]}]}]}

Capacity, fair shares & retry rules

Budget is counted in REQUESTS, not learners. go-scoring gives eval's one credential 30 grade-test requests a minute, speaking and writing together, for every caller of eval — and counts a request once however many answers it carries. eval spends 29 a minute, 10 in flight. Polls (result/{id}) are not counted. How many learners that serves is decided by how you pack answers (measured 2026-09-29):

At most 8 answers per grade-test — the batch size measured to grade identically; a larger one is refused 400. Split it.

Fair shares

The minute's 29 are shared out by X-Sender-Id, so no caller can starve another. Each share is guaranteed; a share nobody is using is lent out. When a send frees, a caller still under its share goes first; otherwise the longest-waiting request does. A sender id not listed has no share of its own and only borrows.

The line

A request past its turn waits in line up to 90 s (longer than the minute, so a burst past the budget gets the next minute's turns; never longer, since Cloudflare cuts eval off at 100 s). Then it is answered 503 with a Retry-After computed from the line — the seconds until the requests ahead of yours would clear, not a guess. A caller that hangs up while waiting leaves the line and hands its turn back; it is never forwarded. A 429 from go-scoring itself reaches you as the same 503 + Retry-After (the wait go-scoring named), never raw.

Slots are fair by caller too: a slot that frees goes to the waiting caller holding the fewest, the longest-waiting among equals.

Retry rules — every caller must follow these

1. 503 + Retry-After: wait that many seconds plus 0–20% random jitter, then retry. At most 3 retries of one request; then give up and grade later (a background job, not the learner's screen).
2. 4xx (bad body, 9+ answers, auth): never retry — fix the request.
3. A poll that times out or errors: poll the same session_id again. Never enqueue the answer again — a second grade-test spends a second turn of the minute for a grade already being made.
4. Poll once a second; stop at status 3 (completed) or 4 (failed).
5. 502/504 (upstream down or stalled): same as 503 with Retry-After: 10, same cap of 3.
6. Send what you already have together. Answers that are ready at the same moment (a whole test, a whole session) belong in one request of up to 8.
7. Your own client timeout: at least 100 s for an enqueue if you want eval's answer rather than your own deadline. A shorter budget is fine — leaving the line costs nobody a turn.

Live counts, per share: GET /health → admission.

Errors

The proxy lanes (/english, /chinese) give an unresponsive upstream 45 s (EVAL_PROXY_TIMEOUT) and then answer 504 (that clock starts once the request holds a slot — see Capacity); a connect or transport failure answers 502. Both name the lane, so an upstream stall is never mistaken for a broken gateway. Keep your own client timeout above that if you want to read the status code rather than hit your own deadline first.