A Matter of Control: The State of Input Device Support in KDE

[todd rme]

If you look at the various changes from KDE 3 to KDE 4, two major
trends emerge: unification and abstraction. Plasma, for example,
unifies the various parts of the desktop and panel. Solid provides an
abstraction layer that hides the details of device management from
applications, while Phonon does the same for multimedia. Akonadi does
both, providing a unified system for handling PIM data and creating an
abstraction layer so PIM front-ends don't need to be concerned with
the source or nature of the data they display. And of course the
success of KDE 4 is not due solely to these trends, it is also due to
developers sitting down and ironing out the current state of the tools
in KDE, where they fail, where they work, where they should be, and
how we can get them there.

Overall this strategy has proved to be wildly successful. Plasma is
now working (at least partially) on everything from cell phones to
finance manager applications. The move from HAL to DeviceKit will be
pretty much transparent to applications thanks to Solid, and the move
from Xine to VLC will be similar for multimedia applications thanks to
Phonon.

However, there is one area that I think has been almost—if not
entirely—neglected in this process. As you can probably guess from the
title, this area is input devices. To clarify, when I say “input
devices”, I mean what some would call “Human Interface Devices”. This
includes physical hardware like mice, keyboards, remotes, and
joysticks, as well as more software-based tools like gesture
recognition and speech recognition. For the most part, the state of
such devices is largely the same as it was in KDE 3.5. kdelirc has
been replaced with kremotecontrol, decent touch-pad support is
available, and Simon may one day provide robust speech control, but
otherwise not much has changed.

KDE, while on the leading edge of most aspects of computing, has not
been for these devices. Some things that have been standard with
pretty much every device or desktop environment for years are
unsupported in KDE. Even areas where KDE is strongest, such as
keyboard shortcuts, there are a number of well-known deficiencies.
Attempts have been made to implement certain features, but they have
been uncoordinated, inconsistent, and redundant.

In my opinion, the issues related to this aspect of KDE deserve some
serious attention, especially with work in many areas of KDE 4 now
focused on polish and incremental improvements. Hopefully this
article, in which I list some of the problems I see and propose a
solution, will get the ball rolling on this front.

I will start by summarizing my take on the state of the major classes
of input devices. The details of the problems are not important, the
point is to justify my conclusion that the issue exists. At the end I
will propose a single solution that I think would solve many of these
issues, one that I feel is based off the successes of KDE 4.
Unfortunately I am not a developer, so the proposal may or may not be
feasible. But at the very least I hope this will at least trigger some
serious thought and discussion on the matter.


Keyboards

I would say that the keyboard is easily the class of input device for
which KDE's support is strongest, one of only two classes that I would
say actually has good support. That being said, there are a number of
issues even here. For example, only one value is allowed for any
global keyboard shortcut, and only two for local shortcuts. This makes
sense for desktop users, but for laptop users who sometimes use an
external multimedia keyboard, they have to decide between using their
multimedia keys or having shortcuts that work while traveling. If you
use Bluetooth remotes, which tend to be identified as keyboard
buttons, this is an even more serious issue since the buttons on the
remote may not correspond to those on your keyboard, meaning you can
only use one or the other. And amongst the multimedia keys my version
of x11 supports, KDE trunk currently cannot recognize 20 of them,
about 13% of the total number.


Remotes

This is the other area where I think KDE support is good, although not
as good as with keyboards. There are two primary problems I see with
the kremotecontrol. First, it requires that the remote already be
configured, not necessarily an easy task and one that generally
requires fiddling with the command-line and editing text files.
Second, it requires an implementation of a shortcut system completely
independent from the keyboard shortcuts. This involves a separate and
very different user interface to accomplish roughly the same task, and
requires that either individual applications or kremotecontrol provide
a second, redundant set of commands. Since remotes are nowhere near as
common as keyboards, this has left kremotecontrol with far fewer
applications it can control directly and far fewer commands it can
issue. It is capable of mapping remote control button presses to D-Bus
events, but this is very complicated and requires the target
application be running.


Mice

KDE's support for mice, or lack thereof, is for me its most glaring
deficiency. For a device as ubiquitous and essential as mice are, they
have gotten remarkably little attention from developers. One of the
more obvious limitations in KDE's mouse support is the lack of support
for now almost universal back and forward buttons. You would be
hard-pressed to find a mouse today, even small, budget, portable,
laptop mice, that do not have these buttons. As far as I am aware
every modern desktop environment supports these buttons besides KDE.
Even Apple, which long resisted even the right-click, now seems to
support these buttons. Many mice today, especially high-end mice, have
many more buttons than this. Besides the normal three buttons+scroll,
my mouse has seven extra buttons, none of which work with KDE. There
is no general way to use mouse button presses or combined keyboard and
mouse button presses for use as shortcuts. These sorts of things are
possible to implement in KDE, rekonq supports back and forward buttons
while plasma supports combined mouse/keyboard shortcuts (although only
local shortcuts, not global ones), but there is no general, KDE-wide
way to use these buttons, the support that does exist is only in one
or two applications, and for most buttons there is no solution in KDE
at all. Feature requests for better mouse button support are amongst
the oldest, highest-voted, and most-duplicated wish-list items in both
bugs.kde.org and the KDE brainstorm system.


Touch-Pads

KDE's touch-pad configuration has greatly improved recently from a
basic configuration standpoint. However, it suffers from the same
limitations as mice, namely that there is no easy way to map touch-pad
events to commands. There are four corners in a standard rectangular
touch-pad, but KDE only lets you map the three mouse buttons, wasting
at least one corner (or more if you have a multi-touch enabled
touch-pad). There are four edges, but only two ways to scroll, meaning
edge actions commonly seen in other touch-pad configurations, like
back/forward and volume control, are not supported. With two
directions per edge and four edges that is eight commands. With
circular scrolling, there are two additional directions for each edge
and each corner, giving 16 additional shortcuts. With multi-touch
enabled touch-pads all of those could be used for non-mouse-related
tasks. That is 28 potential commands on every touch-pad. That many is
probably too much for users to remember at one time, but it is a lot
of untapped capability, much of it capability that has been present in
pretty much every touch-pad driver in windows for at least five years.


Joysticks

Joystick support in KDE is also pretty limited. You can calibrate your
joystick, it shows the current values for each button, and it shows a
trace for two axes (no matter how many axes the joystick actually
has). There may be some joystick controls in some games (although I
wasn't able to find any, even for games where it makes sense).
Ignoring games, however, many joysticks provide a half dozen or even a
dozen buttons and often at least four or five axes, all of which could
be put to good use as shortcuts for controlling your computer. In fact
many joysticks come bundle with tools that let you do this in windows,
and it is probably even more useful for game pads which are smaller,
easier to handle, and often wireless.


Bluetooth

I already mentioned the issues with Bluetooth remotes and keyboards.
Less-standard devices, like Nintendo Wii remotes, currently need
separate software to interpret them, and they have to be mapped to
other input devices like keyboard button presses in order to be used.
There are some efforts to get native support for these sorts of
devices, such as in Plasma Media Center, but once again these
solutions are restricted to parts of KDE and thus will not help KDE as
a whole.


Gestures

KDE has support for mouse gestures. However, the configuration of
these gestures is difficult, relying on the custom shortcut system
which then forces you to map the gestures to keyboard shortcuts or
D-Bus commands.


Speech Recognition

Simon should provide speech recognition support, but since I have
never been able to get it run, and it does not compile at all with Qt
4.7, there is not much I can say. I don't expect the situation to be
much better than with kremotecontrol, though.


A Proposal

As I mentioned at the beginning, I think unification and abstraction
have proven to be extremely successful in KDE wherever they have been
applied. So the questions are, can these approaches be applied to
input devices, and will this make users' and developers' lives better?
I think the answer to these questions is “yes”.

KDE already has a really good D-Bus-based system for handling input
events. This is the keyboard shortcut system I mentioned earlier. It
makes it very easy to assign keyboard shortcuts, and automatically
provides slots for keyboard shortcuts for any toolbar button or menu
entry without any additional effort by developers. However, the
interface would work just as well for shortcuts involving other input
devices. In fact, almost identical user interface elements are used by
Plasma for mouse shortcuts.

So I think the best solution would be to build off the existing
keyboard shortcut system, extending it so developers can write
plug-ins that provide support for additional types of input devices.
The plug-ins would send events to the shortcut system in a standard
format. The shortcut system would then combine the events from all the
plug-ins and pass those events along to applications. Applications
would just see these generic input events, they wouldn't need to know
or care what sort of device sent them (although I suppose this
information could be provided if it would be useful).

This approach has a number of advantages over the current implementations:

   1. Users don't have to map input events to keyboard shortcuts
   2. Users don't need to go searching for the right tool to configure a device
   3. Users don't need to learn several different configuration interfaces
   4. Developers have a consistent software interface for all devices
   5. Application developers don't need to change anything in their applications
   6. Developers don't need to write a custom user interface just for
their device
   7. If one developer programs support for a device the rest of KDE
gets it for free
   8. Support for shortcuts that combine two devices, like
mouse+keyboard shortcuts, would happen automatically
   9. Plug-ins would not need to be tied to physical hardware devices,
so for example a program that triggers a particular event at a
particular time would be easy to do without needing to worry about
cron jobs or shell scripts
  10. Joystick support for games would happen automatically (at least
for buttons)
  11. It should make it easier to keep up with changes in underlying
libraries, like x11 keyboard keys
  12. It would be easy to package and distribute plug-ins
  13. It would be easier to support additional operating systems
  14. Plug-ins could be loaded only when the appropriate device is connected
  15. It could be possible to support multiple plug-ins for the same
device with different features, although this may not be desirable


One objection may be that there are major differences between
different input devices. For instance kremotecontrol supports
profiles, which allows you to change the role of each button. However,
this turns out to be a benefit of this system, rather than a drawback.
It allows developers to focus all their attention solely on the things
that are different about the device they are working on. Each device
would still need configuration tools to configure properties that are
unique to that device, but they can share configuration tools for
things they have in common.

So in the case of kremotecontrol profiles, rather than having to worry
about mapping the button presses in different profiles to different
application events, all the developers would need to do is let the
users assign whatever name they want to the buttons in each profile.
The same name would be seen by the shortcut system as the same button.
If you want a button to do the same thing in different profiles, give
it the same name. If not, give it a different name. If you want
different buttons to do the same thing, give them the same name.

However, you could even take it a step further. Being able to
configure different sets of shortcuts for different situations could
be very helpful, maybe even tied to activities. So the kremotecontrol
developers could, if they wanted, integrate their profile system into
the shortcut system as a whole. That is another advantage of this
system: if there is a useful feature, all devices could benefit from
it rather than just one.

There is one thing that would still need to change in the shortcut
configuration, though. With all of the devices being supported in a
single interface, one global shortcut and two local shortcuts per
command would no longer be sufficient. Users would need to be able to
set an arbitrary number of global and local shortcuts for any command.
As I explained earlier, however, I think this is a worthwhile change
in and of itself, so I don't consider this to be a disadvantage.

In the end, this is just an idea about how the sorts of approaches
that have worked in other parts of KDE could be applied to input
devices. But this is not intended to be the end of the discussion,
rather I hope it is a starting point. A sprint dedicated to input
devices would be ideal, but I think any sort of focused attention on
this area would be of great benefit to KDE. I think that if KDE
developers put their heads together and work some of their magic they
could make KDE's input device support second to none, as they have
done with many other areas.



Note: this was originally a blog post, but some people asked me to
cross-post it here.  The original post can be seen here:

http://neverendingo.blogspot.com/2010/11/matter-of-control-state-of-input-device.html

The blog post has in-line links to various details and support for my
claims.  If you would like me to provide a version with footnotes to
these links I can do so.

I am not neverendingo, he was nice enough to let me use his blog.