HandWritingBench: Turns out, LLMs tend to "write" pretty badly

Most of what we ask computer-use agents to do is click: buttons, links, the odd dropdown. I wanted to see what happens when you ask for something a little more physical, so in this post I ask a question that sounds almost too simple: can a computer-use agent write by hand? Not type the letter. Push the mouse around and draw it.

HandWritingBench is 5,619 letters from 138 writing systems, one letter per task, each graded by a classifier trained for that script. I ran five cheap models on all of it, three times per letter. Every model can write Latin, but almost none can write Devanagari, Thai or Tamil, and the best predictor of failure is how curvy the letter is. The best model, Claude Haiku 5.5, gets 11.5% of letters right overall and 80% of Latin ones. A scripted agent that just traces a font through the same canvas and scorer gets 96% to 99% everywhere, so the canvas and the grader aren't what's failing.

1. The task

The agent gets a prompt ("Handwrite the Greek small letter alpha (α) in the white box using the mouse, like you would with a pen"), a box with guide lines, and a mouse. Whatever is on the canvas at the end gets scored. The canvas ignores synthetic JavaScript events and redraws the scored image from logged pointer strokes, so the only way to get ink down is to actually drag.¹1 None of the five models tried anything else anyway.

Grading crops the ink, skeletonizes it, redraws it at a fixed pen width and hands it to a small CNN for that script. Latin's trains on EMNIST; most others train on fonts, plus human handwriting for the 20 scripts that have a dataset. A benchmark like this is only as good as its scorer, so every scorer gets checked on held-out fonts and writers, on that scripted tracer (the ceiling), and on controls that should score zero, like perfectly drawing the wrong letter (accepted at most 5% of the time in any script).

2. Every model writes Latin. None writes Devanagari.

I ran five low-cost models with thinking off: Claude Haiku 5.5, GPT-6 Luna, Qwen3.8 Flash, DeepSeek V4.1 Flash and Gemini 3.1 Flash-Lite.

Haiku's lead over Qwen is small, 0.8 points, but holds up in a paired comparison (95% interval 0.1 to 1.4). The more interesting thing is the cliff behind the average (Table 1). Every model gets more than half of Latin right, Greek and Cyrillic aren't far behind, and then it falls apart: under 12% on Devanagari, Thai and Tamil, and 5% to 9% across the other 128 scripts.

You might reasonably suspect the Devanagari scorer is just strict. It isn't: it reads held-out human handwriting at 98% and the tracer at 100%. And these aren't near misses, since the right letter is in the scorer's top five guesses only 20% to 26% of the time.

What I find weirdest is that the letter is printed right there in the prompt, in 17-pixel text. So this is closer to copying than to recalling, and they still can't do it. Strokes are smooth and land on the guide lines; Hangul ㄱ, two straight lines, comes out right for four models of five. Devanagari क comes out as a box, or as some other letter. I can't tell from this data whether the models can't make out a glyph that small or just can't turn what they see into strokes. Probably some of both, but it really made me think about how lots of people can speak and read all sorts of languages fluently but can barely write; these agents seem to exhibit something similar here.

The scorer gives one more way to look at this. Its last hidden layer is a decent map of what handwriting looks like, so I measured how far each drawing lands from real people's handwriting of the same letter (Figure 5). In Greek and Cyrillic, a model drawing that gets read correctly is about as close to human handwriting as another person's is. In Devanagari and Tamil, even the couple dozen drawings that pass sit four to eight times farther out than a typical human sample. They aren't messy versions of the letter; they're shapes that just happen to land closer to the right letter than to any other.

A "correct" also isn't always a confident one (Figure 6). When the scorer reads a letter right it's usually sure of it, but in Devanagari, Thai and Tamil the median correct letter gets only 73%, and a quarter of them get under half.

3. What makes a letter hard?

Script-level numbers hide a lot, so I measured a few things about every letter and checked which ones predict failure. The one that wins is curviness: drive a pen along the letter and add up how far it turns (Figure 7); in regards to afety/alignment, considering that human handwriting tends to be curvy, I'm glad this at least tells us LLMs are (for now) incapable of falsifying signatures.

The other candidates were how many people use the script (from Unicode's CLDR population data²2 Population times the share speaking each language, per country, summed over languages whose main script it is. Second-language speakers count, so Latin comes to about 5 billion and Arabic 1.7 billion; 96 of the 138 scripts aren't any living language's main script. ), the fewest strokes the letter needs, and the alphabet's size, which sets the scorer's chance level.

On the least curvy fifth of letters the models average 19%; on the curviest fifth, 2%. The tracer gets 96% to 99% in every one of those bins, so it isn't the scorer. Popularity mostly matters at the very top (57% for Latin and Arabic, 18% for scripts with 100 million to a billion users, 6% to 12% below that), more strokes hurt a little, and alphabet size doesn't matter.

The four measures barely correlate (no pair above 0.15), so a logistic regression can pull them apart.

Roughly, 340° more turning cuts the odds of a correct letter by 55% to 70%, depending on the model. That's a bigger effect than going from a script with a few hundred thousand users to one with a billion.

4. Haiku lifts the pen, Gemini and Luna retrace.

How the letter gets drawn is its own story. Each letter has a fewest-strokes count (H needs 3, N needs 1), and continuity is how close a correct drawing comes to it without going back over its own lines.³3 Stroke ratio (fewest strokes over strokes used) times one minus the share of ink that retraces. It's descriptive here; --continuous asks for few strokes in the prompt and grades it, and I haven't run that yet.

Haiku draws the way people print: separate strokes, lots of tiny mouse moves, almost no retracing. It's the least "continuous" model and also the most accurate. Gemini is the opposite, putting a whole letter down in one or two drags in about three seconds, and going back over five times as much of its own ink.

5. Cost

The full runs cost $28 (Gemini), $38 (Haiku), $41 (Luna), $42 (DeepSeek) and $94 (Qwen, which takes a minute a letter). In case you're wondering: yes—my free credits do not come even close to how much I'd need to test the strongest models available out there (but if you can, please feel free to test them out here or reach out if you'd like to support this project!).

So, can they write?

Sort of. They can hold the pen, and on the script they've most likely seen drawn, they mostly get the letter right. Everywhere else they make clean strokes in the wrong shape, even with the letter printed right above the box. I'd love to see what a frontier model with reasoning does on the bottom of Table 1. The code, data and every run's raw strokes are on GitHub and Hugging Face, --split lite runs a 620-letter version for a few bucks. If you run something bigger on it, please do send me the strokes!

Limitations

• The scorers. 117 of the 138 have only seen fonts, so a long-tail letter written the way people actually write it can be marked wrong. Their confidence is calibrated with a single temperature fit, which is why the log and Brier scores stay secondary.
• The models. Five cheap ones, thinking off, three trials per letter.
• No human baseline. Nobody has written these 5,619 letters through the same canvas with a mouse yet.
• The difficulty measures. Curviness comes from one held-out font per letter, and the population counts include second-language speakers.
• Small samples where it matters most. Only 25 Devanagari and 24 Tamil drawings were read correctly, so the bottom two rows of Figure 5 rest on few points.
• Scope. One letter at a time: no words, no joined Arabic, no CJK.