Apple’s three bilingual speech gains come from two different models

Apple’s new speech paper reports three headline improvements over a bilingual baseline, and its own Table 1 shows they don’t all come from the same model. The 10.4% phone discrimination error and the 72.9% prosody score belong to one intervention. The 56.7% lexical score belongs to a different one.

The paper, Language Discrimination Improves Linguistic Learning in Multilingual Speech Models, went up on arXiv on 27 September 2026 under a CC BY 4.0 licence. Apple added it to its machine learning research site on 2 October. Maureen de Seyssel, Jie Chi and Zakaria Aldeneh trained 95M-parameter HuBERT-Base encoders from scratch on read speech, using Libri-Light for English and Audiocite for French.

What they’re chasing is the multilingual gap. On a matched total budget, a bilingual model given 500 hours of each language trails a monolingual reference given 1,000 hours. So the team tested two ways to push the encoder to tell the two languages apart, holding architecture, language pair and data budget fixed.

The first adds an auxiliary two-way language classifier at layer 6 during iteration one, weighted at 0.3 and ramped over the first 32,000 steps. The second instead replaces the shared 500-cluster target codebook with two language-specific 250-cluster ones.

Both moved the discrimination measure sharply, and by a lot. Language ABX error fell from 27.0% in the baseline to 5.2% with the classifier and 2.9% with per-language targets. Neither intervention needs language identity at test time.

Condition	lang-ABX %	cont. phone-ABX %	sWUGGY %	ProsAudit lexical %
Bilingual baseline, 500h each	27.0	11.6	52.1	68.9
Classifier, iteration one	5.2	10.4	55.2	72.9
Per-language targets	2.9	11.2	56.7	71.9
Monolingual, 1,000h	n/a	10.8	58.5	72.6
Monolingual control, 500h	n/a	11.2	56.5	72.3

Read down the columns and the composite in the abstract becomes visible. The classifier takes the best continuous phone-ABX at 10.4% and the best ProsAudit lexical score at 72.9%, but reaches only 55.2% on sWUGGY, the lexicon-level metric in the ZeroSpeech 2021 benchmark. Per-language targets take sWUGGY at 56.7% and sit at 11.2% and 71.9% on the other two. No single bilingual model in the paper posts all three of the numbers the abstract quotes in one sentence.

But the other figure in that table is the one the write-ups skip. Apple also trained a 500-hour monolingual control on the exact subsets the bilingual model saw, so it has the same per-language exposure. It scores 56.5% on sWUGGY and 72.3% on ProsAudit lexical. The best bilingual lexical result beats it by 0.2 points, which is inside the 1.5 point seed deviation the paper reports for that very condition.

The gap closes against a 1,000-hour monolingual model. Against a 500-hour one, it mostly doesn’t.

Reading of Table 1, arXiv:2609.33345

That’s a narrower claim than “closes the multilingual gap”, though it’s still a real one. The classifier does beat the 500-hour control on continuous phone-ABX, 10.4% against 11.2%, and on ProsAudit lexical. It loses on sWUGGY and on the discrete unit phone-ABX, where the control scores 19.0% against the classifier’s 20.1%. Seed spread that wide across three runs is one of the reasons benchmark numbers mislead so often.

And more data didn’t fix the gap either, which is the part that stings. Doubling the bilingual budget to 1,000 hours per language left sWUGGY at 53.2% and language ABX at 29.0%, slightly worse discrimination than the baseline. A shuffled-label placebo barely moved anything, landing at 51.5% on sWUGGY. Timing mattered more than final strength: applying the classifier at iteration two drove language ABX down to 2.8% but left sWUGGY at 53.0% and pulled ProsAudit lexical down to 66.2%.

Apple’s own framing is cautious, and the paper says so in a single line.

These experiments use public research datasets and standard HuBERT architectures in a controlled research setting and are not intended to describe a production speech system.

Language Discrimination Improves Linguistic Learning in Multilingual Speech Models, Apple

The paper’s limits section adds that the result holds inside an English and French HuBERT setting, and leaves open how broadly it generalises across architectures and languages. Apple has run into that boundary before, when its own GRPO study across 11 languages found both transfer and severe regressions. What’s worth watching next is the segregation measure, which crept from 0.52 to 0.57 as the intervention was sustained, because the useful regime here is discriminable but still shared.