As I have mentioned in 15 earlier blog posts, my current project is putting together an accessibility API comparison framework specifically for browsers, and help weed out substantial differences in implementations.
The challenges I have faced, and still sometimes face:
Windows
I was surprised to learn that Windows is still by far the most popular platform in the world. Interesting. Since it isn’t very POSIXy, I have had a lot of dillemas as how to distribute software for it. The usual autotools, or distools does not have all the advantages it has on Linux or Mac OS X. On any other OS, you could safely assume that Python is installed. Not on Windows. In the last few releases I have been experimenting with py2exe. It gives you a bit of a DOS feel to your package: unzip and run the first EXE you see. Easy, but unelegant.
Selenium
One of the first things that turned me on to Selenium was that it was platform, browser and language agnostic. What better way to provide users with absolute freedom in choosing their language and platform? Originally I subclassed Selenium’s Python client class, and added all of the accessibility functions to it. But I decided it would be awesomely cool if this were added to the Selenium server itself, so that the integration with accessibility would be tight and native, and language agnostic. After installing a couple dozen dependancies, and reading Selenium’s developers guide more than once, I finally made the proper changes in Selenium Core and Selenium Remote Control to support accessibility tests. This created a huge overhead for anyone who would want to build the entire solution from source who is not me. They need to check out my git versions of Selenium Core and Selenium Remote Control, figure out how to build them, fail, try again, etc. Selenium has many advantages, but building it is a real pain.
While Selenium is great, I have had a hard time interfacing with it’s community, besides a few individuals. The changes I made to selenium only made sense if they somehow landed upstream. I don’t see that happening any time soon, so folks who want to use my solution are doomed to using a fork forever. Unacceptable!
So I decided that in the next Speclenium release, I am going to go back to the simple approach of subclassing the Selenium Python client. This will hopefully simplify the setup overhead since we could instruct people to just go to Selenium’s website and download an unmodified binary. The downside, is that tests could only be written in Python, and not in Ruby or PHP. I think that is an acceptable loss.
I already have a branch with the appropriate changes. There will be some dramatic file renaming going on too, not for the first time.
Selenium vs. Windmill
Since I started work with Selenium, I have been called attention to Windmill more than once. For one thing, it seems like Windmill will be the tool of choice for Firefox QA. With components such as MozMill, and Gristmill Firefox QA could do some automated chrome testing action. Recent real-world cases have come up in the Firefox world in which such in-app interaction would be beneficial. Currently the Selenium setup is specifically targeted at the document frame, and nothing above it. This has been the choice since ultimately we are looking for browser independance.
Windmill is written entirely in Python, which would compliment my project well.
So why not switch to Windmill just yet? For one, it would be a distraction, I am pretty far along in the project, I would hate to suddenly switch the whole toolset. Second, I have a feeling that Windmill does not have the same wide support for browsers as Selenium does. For example, Selenium already has the proper bits for automating Google Chrome. I am repeatedly impressed by how easy it is to launch and automate obscure browsers using Selenium.
I know that Windmill has been under intense development this summer, so it might get really good soon. Nonetheless, I think that this accessibity testing project might eventually benifit from a side-by-side approach of support for both Selenium and Windmill.
That is my brain dump for now. Aren’t you glad I don’t do that often?