Smart glasses hear everything around you. A transcript is only useful if it knows who said what, and when. Real conversations are messy: people interrupt, talk over each other, and other conversations happen nearby. Gobi’s previous speech recognition was a single-stream Whisper model. It outputs one stream of words and doesn’t know that different people were speaking.
Why SE-DiCoW. DiCoW (Diarization-Conditioned Whisper) tells Whisper when a given speaker is talking, so it transcribes one person at a time and produces a speaker-attributed transcript. SE-DiCoW adds self-enrollment: it collects a short voice sample of each speaker straight from the recording and uses it to match that speaker throughout the conversation, even when two people fully overlap. These enrollments are also the hook for linking voices to Gobi’s people database later, so a speaker can become a known person.
Deployment. First I reproduced the paper. Then I ran an internal bake-off on three meeting benchmarks (AMI, MMCSG, NOTSOFAR): DiariZen diarization feeding SE-DiCoW beat the single-stream Whisper setup on tcpWER on all three. Adding a voice-activity-detection front end gave about the same result, so I left it out to keep the system simpler. SE-DiCoW now runs as Gobi’s ASR endpoint. It’s asynchronous and scales to zero when idle, which fits after-the-fact transcripts and keeps GPU costs down. It keeps the same HTTP contract as the old Whisper service, so the app only had to change a URL.
What the numbers show. On AMI, a standard benchmark of recorded meetings, my reproduction lands close to the published paper: 21.2% vs. 18.5% tcpWER with automatic diarization. On EasyCom, real conversations recorded from glasses the wearer has on, SE-DiCoW beat Gobi’s previous pipeline (63.5% vs. 73.3%, better on 8 of 10 clips). But the error rate is still far higher than on meetings.
| Setting | Data | Speaker segments | tcpWER |
|---|---|---|---|
| Published SE-DiCoW | AMI-SDM test (16 meetings) | oracle | 14.3% |
| Published SE-DiCoW | AMI-SDM test (16 meetings) | DiariZen (automatic) | 18.5% |
| Gobi reproduction | AMI-SDM test (16 meetings) | oracle | 18.4% |
| Gobi reproduction | AMI-SDM test (16 meetings) | DiariZen (automatic) | 21.2% |
| Previous Gobi pipeline | EasyCom Session_1 (10 egocentric clips) | automatic | 73.3% |
| SE-DiCoW (deployed) | EasyCom Session_1 (10 egocentric clips) | DiariZen (automatic) | 63.5% |
tcpWER = time-constrained, speaker-attributed word error rate (lower is better). AMI rows use Whisper-style English text normalization; EasyCom rows are scored on raw text, so the two benchmarks aren’t directly comparable.
The multi-table test: who is talking with me?
To find out why glasses audio is so much harder, I built synthetic scenes in Unity with text-to-speech. The glasses wearer sits at one of 2 to 5 tables, and every table has its own conversation going at the same time. Only the wearer’s table is the conversation they’re actually in. The audio is clean (no reverb, no noise) and the other tables are always farther and quieter, so this is an easy version of a real café. I evaluated 216 takes.
Example synthetic scene: five tables talking at once, seen from the wearer’s point of view. Captions show what each table is saying; only Table 3 (bottom) is the wearer’s conversation.
I compared two state-of-the-art diarizers, DiariZen and NVIDIA’s Nemotron 3 Diarization, scored two ways with diarization error rate (DER, collar 0.25 s, lower is better):
- All speakers: find everyone who is talking, at every table.
- Attended only (the product setting): only the wearer’s table counts. Speech picked up from other tables is a false alarm.
| Tables | Who is speaking? (all speakers) | Who is talking with me? (attended only) | ||
|---|---|---|---|---|
| DiariZen | Nemotron 3 | DiariZen | Nemotron 3 | |
| 2 | 40.46 | 32.57 | 58.89 | 83.35 |
| 3 | 65.39 | 41.86 | 73.40 | 140.25 |
| 4 | 74.24 | 53.55 | 71.31 | 174.38 |
| 5 | 81.30 | 62.56 | 39.56 | 209.02 |
| Pooled (216 takes) | 71.69 | 52.45 | 59.65 | 163.21 |
The result flips depending on the question. Nemotron 3 is better at finding speech: it wins on “who is speaking?” at every table count. But in the product setting that’s exactly the problem. It faithfully picks up the neighboring tables, and all of that counts as false alarms. Its attended error climbs past 100% (more errors than there is attended speech) and gets worse with every table added: 83 with two tables, 209 with five.
Why today’s best models struggle here
State-of-the-art diarization answers “who is speaking?” Glasses need an answer to “who is talking with me?” The multi-table numbers show that difference directly. The model that’s better on the standard question is worse in the product, because nothing in it knows which conversation the wearer is part of. And since SE-DiCoW transcribes whatever speakers the diarizer hands it, those neighboring-table voices end up in the transcript too.
A simple fix confirms the diagnosis. I added a wearer-table filter after DiariZen that keeps only the speakers who sound close to the glasses. It cut the attended DER from 59.72 to 40.24, almost entirely by removing false alarms (from 44.5% to 17.9%). Deciding which voices matter is the missing piece, and it is a separate research problem from better transcription.
Next: selective listening
Hearing-assistant research shows devices can choose what to listen to. Speech bubbles (Nature Electronics, 2024) mutes every sound outside a set distance from the wearer. UW’s proactive hearing assistants (EMNLP 2025) use the wearer’s own speech and turn-taking rhythm to pick out conversation partners and suppress everyone else.
I’m bringing these ideas to glasses. A Sound Bubble-style distance filter (about 2 m) on glasses microphone arrays is training now, and I’ll compare it against target conversation extraction. No results yet. The goal: keep the people you’re talking with, drop the neighboring tables, and give the ASR only the voices that matter.
Try it: download the Gobi app to use it.
Related reading: Speech bubbles, Nature Electronics · Proactive Hearing Assistants that Isolate Egocentric Conversations (Hu, Itani, Chen, Gollakota, EMNLP 2025) (project page) · DiCoW / SE-DiCoW (BUT Speech@FIT).