Hacking Firefox Mobile, Not The Good Kind Of Hacking

October 17, 2011 //
2

Hi again. Remember my first post to Planet Mozilla? I hope not. Sorry for the male enhancement pills spam. Please stop contacting me for orders.

I am in Toronto right now, spending a week with Mobile folks. It has been about 6 weeks since I started working on a mobile extension, and the list of dirty tricks we employing to get things done is growing. Some of these hacks are there because there is no alternative, and some are there because I just didn’t figure out the right way to do it.

Here is a short list:

Gestures
I reinvented them, poorly. Our extension relies heavily on the ability to support a rich set of multi touch gestures. We have a hack, similar to how it was done in Touching is Good (I know!), where we swap out the callbacks in MouseModule and GestureModule for our own. This doesn’t seem right, for a few reasons. The main one being that this doesn’t seem like a real public API, and it will be broken sooner or later. The second thing is the fact that we are relying on shaky JS to re-interpret the platform-interpreted events. So maybe there is a better way to do things. The nsIDOMSimpleGestureEvent only serves us to a limited degree, I  think there is room for a more low-level multi touch event API that is similar to the Android one. Or maybe a comprehensive high level API with more event types.
Speech
This should probably be implemented natively and have multi-platform support. Right now, in Android, we are doing Javascript <-> js-types <-> jni <-> Java. What could go wrong?! But seriously, it is kind of ugly right now, but I would like to make our JS solution prettier and useful to other extension writers who want to tap in to Android’s API and services.
Security
The Mozilla profile directory in Android is not world-readable, presumably for good reasons. Our extension’s media files need to be world-readable so that Android’s TextToSpeech service could pick them up and use them as earcons. This requires an install and uninstall hook that uses Android’s Context.getDir to get/create a subdirectory in Firefox’s top-level app directory where we could copy over those media files for system consumption. Maybe this is the best way to do it… But I can’t imagine anyone being happy about an extension writing to that directory. Sorry!
–enable-accessibility
Our extension depends on accessibility being enabled at build time. Since I am getting some face time with folks this week, it seems like a good thing to bring up.

I am sure there are other horrible horrible hacks in there. But the list above is really what I needed to confess to. Thanks for empathetic attention.

Categories Accessibility, Software

The Cool

October 13, 2011 //
2

Hello, strange new planet!

There have been countless Steve Jobs eulogies in the past week. Jobs is a complicated figure for me. He joins other historic American innovators such as Bell, Edison and Ford, who’s biographies celebrate the “land of opportunity” mythology, where anyone motivated could get ahead. On one hand they bettered society with affordable mass-produced technology, and on the other hand they employed aggressive business strategies, introduced bad labor practices, and ruthlessly quashed competition.

Dennis Ritchie passed away this Saturday. Eulogies are not competitions, obviously. But the contrasts between Jobs’s and Ritchie’s legacies are hard to ignore. Jobs introduced to the world iconic form factors, gadgets you could hold. But Ritchie and his co-inventors laid the foundation for modern software. That svelte iPad? Its operating system is 40 years old, Ritchie’s brainchild.

But along with the software, Ritchie and his friends introduced an entire philosophy. A philosophy that is just as seductive to an engineer as the latest Apple aluminium unibody product.

I grew up on UNIX. We had a machine at home with the hostname saris, Hebrew for eunuch. My dad taught me The Cool. Specifically the UNIX Cool of keeping it simple, less is more, and silence is success. It is this Cool that made me want to program, and it will outlast every fancy gadget.

My Dad. Still programming, still cool.

Categories Culture, Personal, Technology

Caribou Week Who Is Paying Attention Anyway – We Need a New Maintainer

August 7, 2011 //
4

After making a non-binding resolution to report my Caribou progress on a weekly basis, I flaked. Of course. But luckily Nohemi has picked up the slack and have kept you all up to date about the libcaribou powered GNOME Shell keyboard in her more binding GSoC reports. So no more architecture diagrams are needed, you all get the idea. But if you didn’t, let me make it clear: The goal of Caribou is to make it easy to implement new on screen keyboards where you would only need to provide the view, and libcaribou will be your model and controller.

It is better to admit now than later: I will not have the bandwidth to continue to work on Caribou as my time is slowly running out. So…


What we need:

  • A maintainer.
  • A GTK module, Nohemi is working on this, but we will need similar solutions for other toolkits and fallbacks (XIM?).
  • Unit tests for the library – libcaribou is gaining features and getting complex. We need some tests here.
  • More keyboard layouts/languages.
  • More function keys for the switch scanning keyboard.
  • Finer interaction modes:
    • Modifiers, use latching and traditional key gestures with multitouch.
    • Respect AccessX settings for sticky/slow/bounce keys.
    • A more hardware-like interaction where keys will auto-repeat when held down.
    • etc.
    • … and automatically choose the right mode for the keyboard without comfusing the user.
  • Revisit the “X adapter” and maybe use something more high level.

Anyway, plenty of exciting work. Are you at the summit, please find me if any of this interests you.

Tags
Categories Accessibility, Software

I Am Speaking In Portland Next Week

June 15, 2011 //
0

I'm speaking at Open Source Bridge - June 21–24, 2011 - Portland, ORI am getting really excited about the Open Source Bridge conference in Portland next week. It feels like most Open Source meetups I have attended lately have been on the other end of the Atlantic. It will be nice to be in my element, for once. It will also be nice to meet cool localish people. I am planning to train down there and bring my bike. I hope to see you there!

I’ll be talking about inclusive design.

Categories Accessibility, Personal, Software

Caribou Week Whatever

June 7, 2011 //
1

Hi again.

Since the last time I wrote to you, dear bloggy, I have been working a lot on Caribou, made a trip to welcome Jenny back from Haiti, and crossed the continent by rail.

Let’s talk about Caribou, so much has changed!

Antler

Antler is the Keyboard UI that is bundled with Caribou. It does not have the pretension of being ready for users any time soon, it is more a sample implementation of a libcaribou keyboard,and a place for me to try stuff out and see if it would work in our platform. You could follow Nohemi‘s progress to catch up on how libcaribou is being used to power the new GNOME Shell keyboard UI. With all that said, Antler is still kinda cool. Here is a crappy video of Antler’s touch keyboard in action:

Scanning Support

I redesigned Caribou’s switch support from the ground up with the goal of simple configuration. There is still plenty of more work to do, but after looking at commercial alternatives I feel like we could do a pretty good job. Here is me typing text solely with the right shift key:

Please excuse the green/red combo

Input Method Support (bye AT-SPI!)

This is an experimental tangent, that might or might not be worth the time I spend on it. Recently Caribou master received GTK2/GTK3 input modules that perform DBus calls to the Caribou keyboard, and have it show up when and where it is needed. This has proven to be pretty tricky. I will hopefully follow up with a post about this, and some interesting hacks innovations surrounding these methods. Future work includes writing similar modules for QT3 and QT4. And having the keyboard emit key activation signals that the modules could use for inserting text instead of using XTest which feels so hackish and wrong.

Tags
Categories Accessibility, Software, Technology

Caribou Week 2.98

April 30, 2011 //
2

That diagram I drew up earlier this week? I finished implementing it. It is in the badly named ‘geometry’ branch. I decided to take a day or two and redo some of the C stuff in Vala, with the hope that it will make life easier when I start adding a bunch of classes and create the DBus bits. I regretted it when I found myself hand editing vapi files and amending GIRs generated by valac, but all in all, I am pretty happy with the choice.

If I could be sentimental for just one paragraph and say that all the hard work folks have put into GObject introspection is finally paying off. Big time. Transitioning between all these different languages at will is really amazing. I think we are entering a perfect storm where we have an extremely competitive developer story. There is really nothing quite like this. Yay.

Since I spent this week shuffling deck chairs, I don’t have any exciting new visuals. Hopefully soon.

Tags
Categories Accessibility, Software, Technology

Caribou Week 2

April 26, 2011 //
3

Hello again.

I didn’t get a chance to talk about my plans for Caribou in this little private SOC I am having here. Well here are two oil rigs is a diagram to illustrate it:

Caribou Daemon
The daemon, through AT-SPI and any means necessary figures out when a typing task is needed and activates the keyboard through DBus. When it does this, it provides as much information as possible regarding the text entry task: The location and size of the text area, the location of the cursor and the type of text expected (plain, email, url, number, full name). The daemon is written in Python, at least for now as Python is equipped with the best AT-SPI client library. The daemon has a relatively small body of code, so porting it to C is a worthy mission for some time in the future.
libcaribou
This is a library for keyboard implementors. It is written in C/GObject. It handles all common Caribou keyboard tasks such as:

  • De-serializing Caribou keyboard definition files into KOMs (Keyboard Object Model, I just made that up. You are welcome), with respect to the user’s XKB configuration and keyboard type preferences.
  • Implementing defined keyboard behavior.
  • Exposing the keyboard over DBus with interfaces known to the Caribou Daemon.
  • Implement scanning.
  • Do all the ugly X tasks, XKB, XTest, you name it. This could be replaces/supplemented with a higher level input method in the future.
Keyboard UI
This is the view for libcaribou’s model. It can be written in any language and toolkit that supports GObject introspection. The packaged UI is called Antler, and currently it is less than 200 lines of Python/GTK+. The idea here is that GNOME Shell (or anyone) could easily implement their own keyboard that can take advantage of Caribou’s facilities. If we pull this off well, keyboards will be advertising capabilities and co-exist, so the appropriate keyboard UI is invoked for the given task.

Thanks for reading! In a future post I will describe Antler and plans I have for it.

Tags
Categories Accessibility, Technology

Caribou Week 1

April 22, 2011 //
3

Dear blog,

I have been spending the last while working exclusively on Caribou, trying to get it out of the miserable shape it is in and make sure that it not only remains relevant, but really shines. I have been doing this for two weeks. But the first week was mostly spent on chasing down this bug in GDK. So I really only got down to business this last week.

Here is a photo:

What I have done:

  • Created an introspectable C library, initially to do the keyboard emulation and XKB stuff, but I have grand plans for it. What already landed in master is just simple key synth methods to replace Caribou’s reliance on python-virtkey which was never approved as GNOME external dependency.
  • Created an experimental git branch with a revamp of how we do layout in Caribou. Users will no longer choose between layouts, this is inferred from the current keyboard group the X server is set to. If the user will have any choice it will be between geometries, either a more natural geometry similar to what you have on tablets, or a fuller keyboard emulation.
  • Made “sub keys”, all the latest screen keyboards seem to do this, so now Latin accents and Semitic vowels could be easily entered.

Ok blog, I have to go. I’ll need to write a brief roadmap of where I think we should take this, and how Caribou could play a role in GNOME Shell, etc.

Here is a screencast of what Caribou looks like today.

Tags
Categories Accessibility, Technology

NPR, What Is It Good For?

March 26, 2011 //
0

This, and this.

Categories Personal, World Affairs

GNOME Accessibility Via JHBuild

January 17, 2011 //
3

So a couple of weeks ago I ported Caribou to GTK+ 3 and PyGI. I also stopped testing it (and implicitly stopped supporting it) with CORBA AT-SPI. This means that if you want to download and test Caribou, or any GNOME 3.0 accessibility module for that matter, you need to jump through hoops to get all the prerequisites. There are two sane options:

  1. Install a pre-release distro like Ubuntu 11.04 or Fedora Rawhide. While the Caribou package is not necessarily up to date on these platforms, the prerequisites generally are, so you just need to clone Caribou from git.
  2. Use JHBuild. This is what all serious GNOME testers and developers use. Unfortunately Caribou is not in the current moduleset, and neither are some dependencies. Fortunately, I wrote a small moduleset file that takes the shortest path to having a working version of Caribou for your testing needs. It also has Orca, and other modules will be added in the future. This should probably go into upstream jhbuild eventually.

Once you have installed JHBuild, use the configuration below in your .jhbuildrc, or create .jhbuildrc.a11y, and use jhbuild -f if you have a working jhbuild environment you don’t want to mess with.

build_policy = 'updated'
moduleset = 'http://people.gnome.org/~eitani/a11y/gnome-a11y-3.0.modules'
modules = ['meta-gnome-a11y']
checkoutroot = os.path.expanduser('~/gnome-a11y/source')
prefix = os.path.expanduser('~/gnome-a11y/install')
skip = ['gudev']

Next, run jhbuild bootstrap to install build tools, and install some other development libraries that are worth skipping in jhbuild (for now it’s just gudev).

In Fedora:
yum install libgudev1-devel

In Debian/Ubuntu:
apt-get install libgudev-1.0-dev

Run jhbuild build, and watch things compile successfully.

Now, the hairy part about GNOME accessibility is the fact that there is an AT-SPI registry that needs to start with the session. Optimally the registry runs on a dedicated accessibility D-Bus bus. First you need to make sure you installed and enabled at-spi2 through your distro. On Ubuntu you might want to install it via PPA, on Fedora 14 it already is in the repositories, search for at-spi2. Next you need to enable desktop accessibility and D-Bus support:

gconftool-2 --set /desktop/gnome/interface/at-spi-corba --type bool false
gconftool-2 --set /desktop/gnome/interface/at-spi-dbus --type bool true
gconftool-2 --set /desktop/gnome/interface/accessibility --type bool true

There are a lot of tricks to get the newly built registry to run in your session, but the easiest way is also the most hackish, simply edit /usr/share/dbus-1/services/org.a11y.atspi.Registry.service and substitute /usr with the prefix to your newly installed registry, for example /home/eitan/gnome-a11y/install. Log out and back in again, and you should be using the latest AT-SPI2 registry.

If everything worked, running jhbuild run caribou should bring you the latest version of Caribou.

Good luck!

Categories Accessibility, Software
« Older Entries
Follow

Get every new post delivered to your Inbox.