Go Use It!

The Mozilla Foundation has supported my work on Speclenium, an effort to create a test framework for web accessibility, and a browser benchmark tool, that will allow us to easily spot discrepancies between different browser implementations.
When I originally proposed this project, with the help of Aaron Leventhal, the uses and goals branched out in many directions, so when I finally started work I consolidated the specification to five main points:

  1. It will provide a means of comparison between two static accessible heirarchies of different browsers.
  2. It will provide a means of comparison between two different event sequences with the same interaction in different browsers.
  3. It will run automatically, and will be able to flag discrepancies between two browsers.
  4. There will be a finite set of test cases that will provide a generic benchmark for conformance.
  5. A user should be able to throw it at any URL or interaction script, two browsers, and do a more interactive comparison (without preset rules).

In the hopes of building as much as possible on existing stuff, I used Selenium, a mature automated web test framework to drive the automation, both for launching browsers scripting user interaction. Selenium was perfect because it supported all mainstream browsers, and is cross-platform. By expanding Selenium I made it possible to check for certain accessible objects, subtrees, events or event sequences, both on Linux and Windows.
I also took advantage of the Codetalks test cases that were being written to provide requirement number 4, a set of finite tests.
As for requirement number 5. It took a lot of effort to find a way to make a comprehensible comparison between two complex accessible object trees. Ultimately I think that the end result was worth the effort, which consisted of algorithmic work based on a paper titled Change detection in hierarchically structured information, and of UI work.  You could see the results in my previous post.
I think an important achievement in this project has been the fact that this framework is cross-platform. I did this by utilizing pyatspi for Linux accessibility, and pyia for Windows, the two libraries are similar, and allowed easy cross-platform support.
Pyia is a Python MSAA client library that I wrote early on to get acquainted as quickly as possible with Windows APIs. I got a lot of working code snippets from NVDA when I started working on it. The end result is a fairly straight forward, easy to use library. So if you are planning to do any Windows AT development in Python, I would recommend giving this library a spin. You could download the latest version here.
To conclude, I hope I layed down a foundation for plenty of AT-side testing and conformance benchmarks. I pretty much kept the work browser neutral, so I hope that browser developers across the board will feel comfortable picking this up and using it in their accessibility testing.
A big thanks to Marco, Aaron and Frank for supporting me on this one, and I really hope my work makes a difference.