Why a smaller Whisper model beat my bigger one

Salati has a memorisation mode. You recite a verse from the Quran, the app listens, and it tells you whether you got it right. All of it runs on the phone, with no network, and no recording ever leaves the device. The first version was not good enough: too many correct recitations came back marked wrong.

The obvious conclusion was that the model was too small, so I shipped a bigger one. Whisper large-v3, quantised, just under a gigabyte. The result was not better. It was slower, the first-run download became unreasonable for anyone on a weak connection, and accuracy on Quranic Arabic stayed roughly where it was.

Why a bigger model buys you nothing here

Whisper is trained on general speech across many languages. Quranic Arabic is not the Arabic of news broadcasts or podcasts: classical grammar, a tightly bounded vocabulary, and a recitation style in which vowel length carries meaning. A larger general model is better at exactly what this task does not need, which is breadth. It is no better at what the task does need, which is depth in a very narrow slice.

There is a second problem that has nothing to do with model size. A multilingual model sometimes writes Arabic sounds using Persian or Urdu letter forms. To the ear, ی and ي are the same letter. To a string comparison they are not.

Lever one: tell the model what it is about to hear

In memorisation the expected text is known. The app knows which verse is being practised. Whisper accepts a prompt that goes into the decoder as prior context and shifts probabilities toward those words. It is one line of code.

const { promise } = ctx.transcribeData(pcm, {
  language: 'ar',
  ...(expectedText ? { prompt: expectedText } : {}),
});

Measured across eight real recitations by the reciter Alafasy: word error rate 9.2 per cent without the prompt, 7.9 per cent with it. That sounds small. In effect it is large, because the remaining errors move. Without the prompt the model invents words that are not in the verse at all. With it, almost all that is left are vowel-length errors, and those are exactly what the next step absorbs.

Lever two: compare things that are comparable

Before recitation and source are compared, both pass through the same normalisation. It discards everything irrelevant to the question “was this the same verse?” and folds letter variants together. The order of those steps is not arbitrary.

export function normalizeArabic(text: string): string {
  return text
    .replace(DIACRITICS, '')
    // Map Persian and Urdu forms to Arabic FIRST. Otherwise the
    // [^ء-ي] pass below turns them into spaces and splits words.
    .replace(/ک/g, 'ك')
    .replace(/[یۍ]/g, 'ي')
    .replace(/ے/g, 'ي')
    .replace(/ھ/g, 'ه')
    .replace(/[أإآٱ]/g, 'ا')
    .replace(/ة/g, 'ه')
    .replace(/ى/g, 'ي')
    .replace(/[^ء-ي\s]/g, ' ')
    .replace(/\s+/g, ' ')
    .trim();
}

The comparison itself is a Dice coefficient over words, not exact equality. A single misheard word in a twelve-word verse lowers the score without rejecting the recitation. That leniency is deliberate: someone practising should not be punished for the model's mistake.

The outcome: smaller is better

With prompt conditioning and normalisation in place, model size stopped mattering. The default today is a base model fine-tuned on Tarteel, quantised to q5_0, 55 MB. Beside it, selectable, sits large-v3-turbo at 574 MB for stronger devices. The gigabyte-sized large-v3 is gone.

The quantisation is measured too, not assumed. The same eight recitations produced byte-for-byte identical transcripts with the F16 file and with q5_0. Same accuracy, 63 per cent less to download.

The conversion trap that cost a week

Tarteel publishes its model in Hugging Face format. For whisper.cpp it has to be converted to GGML. The conversion script reads config.json and writes one of its values into the GGML as n_text_ctx.

Tarteel's config.json carries max_length: 1024. That is a generation parameter, not the context length of the text decoder, which Whisper fixes at 448. Take the wrong value and you get a file that looks like a model and that whisper.rn refuses to load. The error message only says the model is unavailable.

Where the weights come from, and why that is provable

The project originally carried a third-party conversion from Hugging Face with no documented provenance. For an app meant to help people memorise a sacred text, that is the wrong foundation. So I convert Tarteel's original myself and host the file on storage I control.

That these are the same weights is not asserted, it is shown: my F16 conversion is byte-identical to the former third-party file, SHA-256 aaebca10…50ead. Here a hash is the entire proof, and it costs nothing.

What I take from this

1. 01Before reaching for a bigger model, ask whether the task needs breadth or depth. This one needed depth.
2. 02If the application knows what is about to be said, that belongs in the model. Prompt conditioning is the cheapest accuracy available.
3. 03The metric has to fit the task. Exact equality was the wrong question here.
4. 04For anything that comes from elsewhere: document the provenance, hash the identity.

The last point had a side effect beyond this project. The letter normalisation is now its own library, because the problem is not confined to Quran apps. Anyone comparing Arabic text out of a multilingual model against a reference source has it.