The Mozilla Foundation has supported my work on Speclenium, an effort to create a test framework for web accessibility, and a browser benchmark tool, that will allow us to easily spot discrepancies between different browser implementations.
When I originally proposed this project, with the help of Aaron Leventhal, the uses and goals branched out in many directions, so when I finally started work I consolidated the specification to five main points:
- It will provide a means of comparison between two static accessible heirarchies of different browsers.
- It will provide a means of comparison between two different event sequences with the same interaction in different browsers.
- It will run automatically, and will be able to flag discrepancies between two browsers.
- There will be a finite set of test cases that will provide a generic benchmark for conformance.
- A user should be able to throw it at any URL or interaction script, two browsers, and do a more interactive comparison (without preset rules).
In the hopes of building as much as possible on existing stuff, I used Selenium, a mature automated web test framework to drive the automation, both for launching browsers scripting user interaction. Selenium was perfect because it supported all mainstream browsers, and is cross-platform. By expanding Selenium I made it possible to check for certain accessible objects, subtrees, events or event sequences, both on Linux and Windows.
I also took advantage of the Codetalks test cases that were being written to provide requirement number 4, a set of finite tests.
As for requirement number 5. It took a lot of effort to find a way to make a comprehensible comparison between two complex accessible object trees. Ultimately I think that the end result was worth the effort, which consisted of algorithmic work based on a paper titled Change detection in hierarchically structured information, and of UI work. You could see the results in my previous post.
I think an important achievement in this project has been the fact that this framework is cross-platform. I did this by utilizing pyatspi for Linux accessibility, and pyia for Windows, the two libraries are similar, and allowed easy cross-platform support.
Pyia is a Python MSAA client library that I wrote early on to get acquainted as quickly as possible with Windows APIs. I got a lot of working code snippets from NVDA when I started working on it. The end result is a fairly straight forward, easy to use library. So if you are planning to do any Windows AT development in Python, I would recommend giving this library a spin. You could download the latest version here.
To conclude, I hope I layed down a foundation for plenty of AT-side testing and conformance benchmarks. I pretty much kept the work browser neutral, so I hope that browser developers across the board will feel comfortable picking this up and using it in their accessibility testing.
A big thanks to Marco, Aaron and Frank for supporting me on this one, and I really hope my work makes a difference.