Field notes · Japanese speech recognition
How we establish data quality.
One character changes the number of tablets a speaker reports taking. A physician can correct that error for training. Measuring whether the correction helps on another voice takes a separate recording and an independent reference.
- Model draft
- 二錠飲みました。I took two tablets.
- Physician correction
- 一錠飲みました。I took one tablet.
Evaluation design
Follow one correction
1. Check the recording
Before an encounter enters a dataset, confirm permission for the intended use, who may hear it, and which identifiers must be masked.
Trace the fictional dose phrase to its authorized source and inspect what an export contains. Mozu requires real clinical recordings to remain in the approved environment in Japan.
2. Correct the training draft
A physician listens to the audio while reviewing the model draft. In our example the speaker says one tablet. Correcting the tablet count gives training a label grounded in what was said.
Compare the audio, draft and correction. Record unresolved audio and the time spent reviewing it.
3. Build an independent reference
For a held-out encounter, two physicians would transcribe from the audio independently, without seeing the model draft or each other’s text. A third would listen where they disagree.
If their versions differ on the tablet count, preserve both readings and the adjudicator’s decision. Measure disagreement before adjudication. Keep this reference out of training.
4. Hold out entire physicians
Assign every encounter from a recorded physician to one set only. Use validation physicians to choose settings. Reserve test physicians for the final comparison.
Audit the physician lists for overlap. To test tablet counts on unfamiliar voices, use separate held-out utterances and report how many physicians and dose mentions they cover.
5. Compare the learning curve
Start with the unchanged model at zero reviewed hours. Retrain from that base at planned totals of 50, 100 and 150 hours, and report actual hours retained at each checkpoint.
Score every checkpoint on the same fixed references under the same decoding rules. Count wrong doses separately: fewer errors elsewhere can lower the overall character error rate while the tablet count is still wrong.
Evidence to date
What has been tested?
- Current workflow
- The review interface supports listening to audio alongside Japanese transcript text. A local check exercised that workflow with synthetic material.
- Pilot observation
- Mozu’s team ran a pilot using 12 mock consultations. It reported instances of the model substituting a more conventional phrase for the spoken wording. The pilot was small and affected by two implementation bugs. Its results leave the effect on real consultations open.
- Proposed experiment
- The design targets a fixed 20-hour golden test set and a separate five-hour validation set. The physician split, independent references and training-hour checkpoints are design targets. Their completion and any gain on real encounters still need verification.
For AI labs
What to ask for in an evaluation
- Coverage
- How many distinct physicians appear in each set? How many held-out examples contain a drug name, a dose or a negation? Give counts alongside percentages.
- Paired comparison
- Show the base and trained model on the same consultations. Report the difference and an uncertainty interval, for example by resampling whole consultations together with both models’ scores. Check whether one physician drives the result.
- Errors that change meaning
- Count missed or inserted drug names, wrong numbers, reversed negations and speech invented during silence. Show the number of opportunities for each error.
- Review effort
- Compare physician review minutes per audio hour on similar encounters. Report corrections still needed and disagreement between independent transcribers alongside model accuracy.
Discuss the evidence for your task.