Starship
Ktav

94% character accuracy, and the notes are still unreadable

What a good-looking metric hides when the unit of value is a sentence, not a letter.

I am training a model to read handwritten Hebrew lecture notes. Mine, mostly, which gives me a labelled set nobody else has.

Character accuracy is 94.1% on a held-out set of 2,400 characters. That number made me very happy for about a day.

Then I read the output

A Hebrew word averages 4.2 characters in my notes. At 94% per character, the chance of getting a whole word right is 0.94^4.2, which is about 77%. So roughly one word in four is wrong.

A sentence of twelve words comes out clean about 5% of the time.

The metric was measuring the thing I could count, not the thing I needed.

What changed

I switched the evaluation to word-level accuracy and the whole picture reorganised itself. Errors cluster: final-form letters, and anything written after about 40 minutes of a lecture when my handwriting degrades. Two specific problems instead of a vague 6%.

Word accuracy now: 81.3%, up from 77%, by oversampling late-lecture pages in training.

What I have not solved

Reading someone else's handwriting. I have one volunteer's notes and the model gets 41% of words. A set of one person is not a dataset, it is a personal tool with ambitions.

3 reactions

One project a week, in your inbox

What was built, the numbers it produced, and what did not work. No other email, and one click to leave.