[GSoC] New proposal about content interaction improvements

[Emmanuel Lepage-Vallée]

Sorry to be a little late, but here is my proposal for this year. You can
find it on the GSoC project page, but for those without account, I link it
here. It is a new idea and was prototyped during the last few weeks. It may
be a little radical for a GSoC project, I am taking some risks, but as I
think it would a nice fit in KDE Library and follow the KDE Project spirit
of innovation are shared modules across application, I submit it with the
hope it will get accepted. I talked a little about it on IRC and KDE Project
forum (
http://forum.kde.org/viewtopic.php?f=27&t=87007&sid=06ed06aa7af8ea8728ad104cea73c3bc)
to get come retraction, but I am now posting the full text here as all I
cold get is "it would be useful" and comment like those.

Please comment my proposal, my ideas and what need to be adjusted in order
to get them in KDELibs (as I am aware some concepts are quite vague, even if
I do my best to explain how they work).

*The proposal:*


Hi, my name is Emmanuel Lepage Vallee, a french Canadian student living in
Quebec city. The is my third proposal. The 2008 one was about plasma media
center, but KDE 4.0 had higher priority problems to deal with. My second
one, in 2009, was about Plasma Netbook, but Marco got a paid job to do it
and some other factors ended in the dismiss of my proposal. I was not sure
if I would submit one this year, it is why I am a little late. But, as a KDE
(SC and Library), user, I really want my user experience to get better. This
is the subject of this proposal. I love KDE and use it since the 2.x era and
know why it is the best software compilation ever. KIO, KAction, KParts,
XMLGui, KJobs, Services menu and all these features make my (geek) life so
much easier. We enter a new decade now, competitor want to push new concept
(3D-ui, finger gesture, mobile devices and tablets to name a few) and like
it or not, KDE need to follow, or even take the lead in those area. For me,
Nepomuk, Plasma and Plasma data engines are the first key elements of this
evolution (in KDE). It is why it was at the center of my two previous
(refused) proposal.



 This years, it will be different. In the last two weeks, I was studiing
wishes, bugs and use cases from various origins to see what KDE could do
better. Strangely, one item came ahead, one that I did not expected to see:
KActions. We take them for granted, but are they really as good as they can
get, is it sufficient? I discovered that, no, they are not as perfect or
even good enough for most use case. One of my coworker (I currently work as
a Qt consultant to make money for college) use the Plan9 desktop, and
actions were radically different in this (dated) desktop, but I looked at
him work for a few minute and understood why he was using that “thing”. Then
I came across other use cases and in the end, they needed some improvement
to this system. I also came across few other details across KDE base (or
close to it) where I would want to work a little.



 I code a lot of prototype, proof of concepts and so on, but never have time
to finish them. I have to pay college and all the real life activities, it
make me sad to never be able to push anything to the trunk, even if I have
my SVN account since quite a while. I have all that code around me that need
some love to be pushed, it is why this proposal is closed to a “todo” list
than a project. I have more than 30k lines of unsubmitted code on my
desktop, ranging from Konsole KPart replacement, WYSIWYG Web editor, Adobe
flash creator, Okular improvements, a few Plasmoids and so on. But this
can't be a GSoC project, so let's keep it simple. Having time where I can
freely work on KDE would be great for me, really. If this GSoC project is
accepted, I will work on it at full time and enjoy finally having my chance
to work on KDE without having to worry about where I will sleep the next
month if I do.



 My project would be to work on KDElibs to lay a basis for further
improvements. As I said, KDE have one, if not the best unified architecture,
and some work may help leveraging it to a new level, so we can continue to
build on it. The stronger is the base, the better will the software get. We
can see this improvement as KDE applications slowly rewrite KDE3 code to the
pillars of KDE4. I want to extend those pillars, without braking the API,
ABI or anything to add missing features to fill the use cases I collected in
the last few week. The improvement can be separated in 3 caterogies:

   -

   Improve the KAction system to enable multitouch on current applications
   (on OS that support it, of course) and make drag and drop more intuitive
   across applications
   -

   Improving the KDE text widgets (KTextWidget, KTextBrowser and KatePart)
   to be a little smarter by taking examples from the services menu framework
   -

   Work on XMLGui to restore some of the most missed features from KDE3
   while making the concept stronger so it can be extended even further

As you can see, even if they are not directly related, those point share
many aspects. First, they are targeting direct user interaction with the
content. They are also all building blocks that existing applications will
automatically take advantage of with little or no work at all. New features
without loss of performances, duplication across applications or slower
start up time.



 Improving KActions

The first point is also the faster to implement, in fact, half of it is
mostly ready. The API need to be approved and tested against more use cases.
Current actions are only buttons or checkboxes, they are not active and can
not really be aware of the content they are executed against. Some are less
or more, like copy, cut and paste, but they can not work on object or
content itself, they apply to the whole application. The first idea, taken
from the AT&T Bell labs Plan9 operating system, is to apply actions on
content instead of applications (when it apply, of course). Most of the
current applications involving selection in some form or an other (text,
images, files) execute the action then take a look at the current selection
(like most applications since WordPerfect). But what if it was the other way
around: you pass the selection to the action. Immediate advantages are
hybrid buttons/menu items. Where current action have a certain behavior, but
some step may be skipped if you pass directly what you want to do. The best
example opening a file. When you press the “open” action in KStandardAction,
it open a dialog. But if you drag and drop the file or middle-click-paste
the file path directly on the action button? In that use case, the user know
exactly what he/she want to do, so opening a dialog is a waste of time. The
action dictate the result directly. Much application already implement some
kind of support for this, but it could be better if it was associated to an
action instead of the whole application. If I drop a file on Krita, I expect
it to open the file. But if I drop a file on the “new” item in the toolbar,
I would expect it to create a copy of the content.



 By enabling this behavior in KAction, every KStandardAction would
instantaneously be able to take advantage of it, without any new code in the
applications themselves. I know this is hard to understand first, you have
to see someone work like that in front of view to really understand how it
work and why it work better. I have made a little proof of concept of this.
It does not in dept as it only took me 3 hours to implement the most basic
framework (then 10 minutes to add a few use case prototype). What you will
see here is not hardcoded, only the use cases are. I extended KToolBar,
KAction and KToolButton (new class) to make it work. It does not break the
API nor add too many new functions (5). Just enough to handle drag and drop
and middle click paste. It also add both signals when this happen (QVariant
and QMineData).

*It show most of this proposal concepts, most of them will be explained
later:

http://www.youtube.com/watch?v=U9PrznRoPWc

As you can see, it work quite well and does not affect the look of the
toolbar, it just add the possibility to interact with actions using
arguments. I repeat myself, but as it is important: it is 100% compatible
with the current API it just add some features.



 My second feature on KAction would be to add mouse gesture and multitouch
support. Mouse gesture can be tried by using the Firegesture plugin in
Firefox. It allow to connect action to movements of the cursor. Multitouch
is similar when the OS support it (not X.org, but it may change soon).



 Current actions support two keyboard shortcuts. It's good, but is useless
on anything with a touchscreen, as user will always try to use the
touchscreen instead of the keyboard, making the “power user” keyboard
shortcut inefficient instead of more efficient than buttons (what they are
supposed to be). By adding a third and fourth shortcut for those kind of
input devices it would allow KDE Applications to take advantage of the
touchscreen without too much effort from applications developper. Linking
KActions to gesture element is the easiest path to have a decent multitouch
support. It is not perfect, per widget multitouch will still be needed in
some applications, but this feature would add a strong multitouch support at
no cost. Let's imagine few simple use cases:

1:

Robert have an EeePC with a touchscreen mod and a few keys around the
screen. Even without a multitouch screen, he can match one of the key around
the screen plus a gesture to switch channal in Konversation. So pressing the
control button and moving the finger to the left would switch to the
previous IRC channal and doing so to the right will display the next
chatroom.

2:

Dylan own a multitouchscreen on his brand new tablet. Linux just got support
for multitouch as a summer of code project for X.org. He is using two finger
gesture in ReKonq/Konqueror: |__ and __| to go back or forward in history
and ¯¯| and |¯¯ to switch from tab to tab. He also use three finger gesture:

← and → switch across windows and up and down to switch virtual desktops.
Like that, he does not have to click on tiny buttons with his finger on the
edge of the screen to trigger the change.



 In both of these use case, enabling those applications (and KWin) to take
advantages of touch sensitive hardware ( or mouse junkies ) would not
require any modifications to the code as they currently support those
actions as keyboard shortcut. A gesture editor would be needed (in the
shortcut editor dialog) and temporarily catching mouse/touch signals in
KMainWindow or KXmlGui then trow them back in if it is not for us. It would
not damage performance that much and could be disabled if no gesture is set
for a particular application.



 I don't have any multitouch hardware (other than a iPod touch), but I have
an EeePc with a resistive touchscreen and plan to buy a N900 soon. So I will
be able to test the mouse gesture / non-multitoutch feature, but not the
multitouch one (I may buy one of these hybrid notebook/tablet from Lenovo to
replace my aging laptop if I really need a test platform).
Improving KDE text editing

My second feature is about KTextEdit. As you just saw in the video above. I
would want to add an extension system based on the current text selection
using regular expression. This is once again an idea taken from the Plan9
operating system, but also from KDE, as file already have this feature
(services menus). There is two distinct subcategory to this feature: the
context menu and active text.



 The context menu is definitely the most easy to discover and use. As shown
in the proof of concept, simple regex matching allow plugins to trigger
various action based on the selection. They can alter the content, process
“plus value” out of it or ask other applications to take on and do the
actual job. This use the KAction extension system as shown above, as it is a
better way (even more in this case) to handle selection. It is also much
more secure, as the plugins could be sandboxed and not able to access the
file itself as the *copy* action actually do. Of course, this would not
replace the current context menu as it does in the video, but append new
actions. With Microsoft Office 2007-2010 using context menu more and more, I
think people will start to use them in other applications, even if most user
currently don't (in fact, I think that over 50% of computer user don't even
understand what a context menu is).



 The second type is active plugins. Similar to macros in office documents.
In the video, I use a really bad example when I turn text into toolbar. It
may be cool if you want to create your own toolbar and use a splitview
application (like Kate), but otherwise is totally useless. I used this
example because it was easy to code and I don't want to spend time on a demo
while the actual framework is not compatible with it. As I said in the
video, this would work better with file with a special syntax, so
macros/plugins can parse the content and react with the user working on the
document itself. It could be used for plugin based autocompletion, syntax
checking, text correction, translation, refactoring, turning a language into
an other (it: man page into HTML, RSS into HTML) and so on. As seen in
Firefox when JavaScript developers have access to data, they can be quite
imaginative how to use the framework to deliver new features.



 Usages are limitless and could even be declared in the file itself (with an
OpenOffice like macro warning, of course) so a text file can become a micro
application by itself, with nothing else. To do that, I need five things: my
KAction extensions, Qt Script/JavaScript interpreter, an API, DataEngines
connection and KGetHotNewStuff to make it easier for user to get scripts. As
much of these script does not need dependencies (even if the should not be
forbidden), KGHNS would be a good delivery system.



 The API itself is not yet designed, and probably would not be frozen until
KDE 4.7 (with a preview in 4.6). We have to avoid the Plasmoid desaster.
Plasmoids can be written in many languages: some compiled, some interpreted
but not installed on the user computer. It make it really hard for users to
have an easy to access source of extensions. Pure JavaScript/QtScript
extensions should make it easy to write and easy to deploy, even if some
devs prefer python or C++.



 Adding support for those plugins in Okular, KOffice, KDevelop (they have a
similar system in 4.x), KHTML/KdeWebKit and KTextBroser (new class?) could
be done later.
Improving XMLGui

The last item is KXmlGui. I don't want to reinvent it or add crazy new
features, just extend it a little. First, restore global menubar and make a
Plasmoid for it. User want it, so let's give them. I would less or more copy
the Gnome D-Bus API so it can become a FreeDesktop specification. It would
also be easier to make GTK apps work in KDE global menubar applet. The
Bespin way of doing it is not really good, and does not work right. Adding
support directly in KXmlGui will be easier and more flexible, but it will
not work on pure Qt apps. I would also add support for toolbars, so
application can share toolbars space. It is good when you have tilled window
manager, and as KWin will soo add that feature, then it make sense to add
global toolbar to Plasma. Many toolbars could be added to the toolbar
Plasmoid like you can currently do in one application, but those toolbars
could come from different applications. It's just a question of designing a
good d-bus API.



 One other little feature I would like to add is the ability to design menus
and context menu like you can currently edit toolbars and shortcuts. It
would be integrated with the KTextPlugin 2.0 framework explained above. So
user can match a context menu with an use case (or directly to a RegEx).



 The last thing I would want to add is the ability for text plugins to
register an action in toolbars and menus. It would use the two previous
features (Extended KActions and text plugins).





 That's it! It is what I would like to contribute to KDE SC this year. It
may seem a lot, but I have already some code and the necessary experience to
do it. Both API (dbus and QtScript) would be designed during the GSoC
project so I wont talk about them too much in the proposal. I think this
should be a solid contribution to KDE. Even if it is just a bunch of little
details, it is in the spirit of KIO and services menu. Not a killer feature
by itself, but when used widely in KDE applications, it give them so much
more value. This proposal is a lot shorter than my previous one, and have
been put toughener in less time than I used to invest in them. I have
learned from the previous two, spending 100 hours on the proposal wont make
you in, so I try this one. It is just a bunch of idea and wishes grouped
around a central theme. An incremental addition on an already strong code
base. I have the skill to do it, I code for KDE since 3.x, but as said
earlier, I rarely submit anything because it does not match my own quality
standard (and they are quite high). I need to take my time and come out with
something good. I don't have the time to do it on a table corner between two
contract or classes.



 The timeline and objectives would look like (in order):

1.5 week: Finishing the new KAction framework

2.5 weeks: Adding features to KXmlGui, designing and proposing the d-bus API
to review

1 week: To complete a working RegEx based selection interpreter to lunch the
right plugin

3 weeks: Create a working extension framework that could be a preview in KDE
4.6.

The final and complete version (of the new KTextPlugin2 engine), with access
to dataengines and other modules of KDE will be written during the rest of
the summer to be ready for KDE 4.7 in July/August 2011. After that release,
the API could be tagged as stable future versions would be backward
compatible with that API. Pure QtScript plugins with only the ability to
receive and alter selection (context menu script) and read the document and
react on basic events (macros like scripts) are going to be part of this
GSoC.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.kde.org/pipermail/kde-core-devel/attachments/20100406/11830441/attachment.htm>