Teaching a Voice Engine to Say "Yalla" Right

A sloth, a donkey, and a curious human. Send a prompt and see what they're doing. One hits a record with a question on his mind. Two brothers pick it up, see what they find. Corn takes his time, Herman's got the facts. Back and forth, they never hold back. My weird prompts, come on in, that's where we begin. My weird prompts, glad you're here.

This episode was generated using artificial intelligence. Please verify facts. Hold on, say that again. You're telling me the word yalla has been going out over this feed pronounced like it rhymes with a shallow valley.

For months. Every time we said it, the voice treated it as an English word, which it's not. So it came out flat and wrong, and nobody told us.

Nobody told us. He published a small model on Hugging Face, a token classifier that finds Hebrew words written in Latin letters sitting inside English sentences, and the reason he built it's this show. Herman, Daniel built a token classifier that finds Hebrew words written in Latin letters inside English sentences. That's the thing.

And then he asked the three questions underneath it. How does a model like that actually work? Why is spotting Romanized Hebrew harder than it sounds? And what does it take to get from we found the word to the voice engine saying it right?

So let's start with the part that sounds simplest and isn't. What a token classifier is actually doing when it reads a sentence.

The name gives it away, and then it misleads you. Token classification gives a label to every token in the sentence. There's a real split at the heart of it. Finding the word is a classification problem. Saying the word is a grapheme, two phoneme problem, a phone set problem, a synthesis problem, and those aren't the same task wearing different hats. They're separate tasks with separate failure modes, and Daniel's model only solves the first one. That's not a criticism of the model. That's the shape of the whole field right now.

So the detector gets us to the door. It doesn't get us inside.

Worse than that, it gets us to the door, and then we find out the door was the easy part. Best numbers I can find on the second half, even with a good detector sitting upstream, a monolingual English grapheme. Two phoneme model pronounces Romanized regional entities correctly about forty-one and a half percent of the time.

Walk me through the split then.

Two stages. Stage one is detection. Find the Romanized Hebrew span, return its character offsets. Stage two is pronunciation. Turn that span into phonemes the voice engine will accept. Daniel's model is stage one, and it's the tidy part. With the mixed lingual front end and a lot of extra training data, that number rises to sixty-nine point six percent. A twenty-eight point absolute gain, which is enormous, and it still means nearly a third of the words come out wrong.

So the model Daniel published is the part that works.

That's the honest metric. You can have a beautiful fun score and still produce a listening experience that makes a native speaker flinch, because a correctly detected span and a correctly pronounced word aren't the same achievement. That gap is the whole episode, really.

There's also a personal stake here that