I thought testing with VoiceOver couldn’t be that hard

Over the past few months, I’ve been getting deeper into accessibility. WCAG, BFSG, contrast methods, semantic HTML, the differences between WCAG 2.1 and 2.2 — you name it. And I kept hearing the same thing from accessibility experts: if you want to know whether something really works for blind users, test it with blind users.

My reaction was basically: sure. Makes sense. But how hard can it really be to test a technically accessible website with VoiceOver myself?

Luckily, VoiceOver is built into Apple devices, so there was no special setup standing between me and finding out. I tried it first on my Mac and then on my iPad. The website had passed the usual technical checks, VoiceOver itself is not impossibly difficult to operate, and I’m very comfortable with audio. I’d rather listen to a podcast than read a magazine, and audiobooks beat printed books for me by a mile. Still, what came out of the speakers mostly sounded like Minionese. Sadly, the word “banana” never appeared.

My first thought was that the visual information might be getting in the way. Maybe I was unconsciously relying on the screen instead of properly listening. So I put on a sleep mask and tried again. That did not suddenly turn the experience into something meaningful. Now I simply had Minionese without the screen.

That was the point where I realised what I had underestimated. The problem was not audio. I was missing the mental model. An experienced blind screen-reader user has routines, expectations and strategies for navigating headings, regions, links, forms and controls. They know what matters, what can be skipped and how the pieces fit together. I didn’t. I could technically operate VoiceOver, but I could not make proper sense of the experience.

That hit me because my job is, in many ways, to look at products with an expert eye. Experience, heuristics and pattern recognition help me spot problems quickly, and expert accessibility reviews can do the same. But when your own context is miles away from the actual usage context, there are limits to what that expert view can tell you. You can learn, practise, build empathy and become very good at testing with assistive technology. You are still working with a model of somebody else’s experience.

I had assumed that with enough technical understanding, a bit of VoiceOver practice and some empathy, I would be able to judge the experience reasonably well. I couldn’t. And I now understand why accessibility experts keep insisting on testing with actual blind users.

Turns out it really was that hard.