94% character accuracy, and the notes are still unreadable
What a good-looking metric hides when the unit of value is a sentence, not a letter.
I am training a model to read handwritten Hebrew lecture notes. Mine, mostly, which gives me a labelled set nobody else has.
Character accuracy is 94.1% on a held-out set of 2,400 characters. That number made me very happy for about a day.
Then I read the output
A Hebrew word averages 4.2 characters in my notes. At 94% per character, the chance of getting a whole word right is 0.94^4.2, which is about 77%. So roughly one word in four is wrong.
A sentence of twelve words comes out clean about 5% of the time.
The metric was measuring the thing I could count, not the thing I needed.
What changed
I switched the evaluation to word-level accuracy and the whole picture reorganised itself. Errors cluster: final-form letters, and anything written after about 40 minutes of a lecture when my handwriting degrades. Two specific problems instead of a vague 6%.
Word accuracy now: 81.3%, up from 77%, by oversampling late-lecture pages in training.
What I have not solved
Reading someone else's handwriting. I have one volunteer's notes and the model gets 41% of words. A set of one person is not a dataset, it is a personal tool with ambitions.