Review Request 129727: kdeconnect-kde: Add remotekeyboard plugin

Fri Jan 13 20:42:16 UTC 2017

> On Jan. 10, 2017, 9:35 p.m., Albert Vaca Cintora wrote:
> > >For the json-payload I used the syntax of the key-events as sent by mousepad-plugin with the addition of a "sendAck"-flag. If "sendAck" is set to true the remote peer should echo a key-event if it could be handled, thus allowing the local client to find out whether the key was accepted. For performance reasons, it's allowed to send multi-char strings in the "key" property (performs much better if you send a whole string via "echo '...' | kdeconnect-cli ..." e.g.)
> > 
> > I like the fact that you used the same protocol. This actually should allow us to send keys phone-to-phone and kde-to-kde, but as of now it's not working because the package type used phone-to-desktop are called "kdeconnect.mousepad" and the desktop-to-phone are called "kdeconnect.remotekeyboard". We tried changing your code to use "kdeconnect.mousepad" and it worked Android to Android :) so it might be a good idea to rename the package type (even though it only implements the keyboard part and not the cursor part, but it's not a big deal). It's true that you "extended" the protocol by adding the "echos" (maybe they can be renamed to "kdeconnect.mousepad.echos" to make it more clear?) but I don't think it will break if echos are not present (eg: desktop-to-desktop).
> > 
> > >kdeconnect-cli: For now takes a string and transforms it into single key-events for visible characters only. In a first implementation I used a kbhit() helper that used termios.h to catch and relay keypresses interactively (including some special-events), which was not optimal. A better approch might be to use linux input-api directly. Would this be an option regarding cross-platform compatibility or can I assume to develop for Linux only? Being a command-line guy, I'd really like to have a fully featured kdeconnect-cli interface ;-)
> > 
> > I like your approach because it's simple and works multi-platform.
> > 
> > >Factor out the Qt::Key-to-internal keymap to some core-helper because it corresponds to the mapping in the mousepad-plugin?
> > 
> > It sounds like a good idea, we can do it in a separate patch.
> > 
> > >The plasmoid is not perfect as it is: A single line containing a non-echoing TextField (i.e. it eats all the KeyPress events), and only ack-ed keypress-packets from the peer device are injected if they contain visible keys. Advantage: the user sees whether his key-presses are accepted by the peer device. Disadvantage: The echoed text does not correspond 1:1 to what is shown on the peer's display, user might be confused when typing without success. I played around with different variations each of which with its proper shortcomings:
> > 1. An echoing Textfield for typing: Has the advantage that the user can directly see what he is typing, which makes interaction in the typing field easier, BUT messes up interaction if the Editor on the peer is changed silently and does not notify the user if his keypresses are not handled by the peer.
> > 2. A non-echoing TextField for typing PLUS a readonly one for printing visible echoed keys. Disadvantage: same as for the previous one and uses more space on the plasmoid.
> > Comments? Ideas?
> > 
> > Agreed. I also found it a bit confusing that it is no 1:1. Maybe if we echo a few more things like the Backspaces it will get better?
> > 
> > 
> > One extra thought: right now the Plasmoid doesn't know if the keyboard is opened on the Android side or not. If we make the Android side notify the desktop when the keyboard is open or closed we can use that info on the desktop. For example: gain focus of the input field automatically, or hide it when there is no receiver on Android. Any thoughts?
> 
> Holger Kaelberer wrote:
>     > We tried changing your code to use "kdeconnect.mousepad" and it worked Android to Android :) so it might be a good idea to rename the package type (even though it only implements the keyboard part and not the cursor part, but it's not a big deal).
>     
>     Ok, will change it to mousepad!
>     
>     >> The plasmoid is not perfect as it is: A single line containing a non-echoing TextField (i.e. it eats all the KeyPress events), and only ack-ed keypress-packets from the peer device are injected if they contain visible keys. Advantage: the user sees whether his key-presses are accepted by the peer device. Disadvantage: The echoed text does not correspond 1:1 to what is shown on the peer's display, user might be confused when typing without success. I played around with different variations each of which with its proper shortcomings:
>     1. An echoing Textfield for typing: Has the advantage that the user can directly see what he is typing, which makes interaction in the typing field easier, BUT messes up interaction if the Editor on the peer is changed silently and does not notify the user if his keypresses are not handled by the peer.
>     2. A non-echoing TextField for typing PLUS a readonly one for printing visible echoed keys. Disadvantage: same as for the previous one and uses more space on the plasmoid.
>     Comments? Ideas?
>     
>     > Agreed. I also found it a bit confusing that it is no 1:1. Maybe if we echo a few more things like the Backspaces it will get better?
>     
>     Not sure, unless we interpret it completely as is done on android side it will feel incomplete. My first try was to inject a real QKeyEvent upon receiving an echo package into the Textfield, which would result in a (almost) perfect behavoir as the Android keyhandling tries to mimic TextField's handling of special keys on desktop. I tried hard via a native wrapper for QCoreApplication::postEvent() but there was never a event arriving at the QQuickItem. Hints from the Qt-experts?
>     
>     Another option would be to simply disable echoing in the plasmoid completely and let the user look at the result on the Android side.
>     
>     > One extra thought: right now the Plasmoid doesn't know if the keyboard is opened on the Android side or not. If we make the Android side notify the desktop when the keyboard is open or closed we can use that info on the desktop. For example: gain focus of the input field automatically, or hide it when there is no receiver on Android. Any thoughts?
>     
>     Thought about an extra package for that case, too. For now the echoing in the plasmoid is supposed to give the user feedback about whether his keypresses are handled remotely or not. We could add a kdeconnect.mousepad.keyboard_status package and act as you proposed (gray-out vs. activate the textfield or so). With this feedback keypress echoing could also be disabled in the plasmoid completely because we give other feedback to the user.
> 
> Albert Vaca Cintora wrote:
>     I like the idea of giving some other feedback instead of trying to mimic the input field in Android... the later seems too hard to achieve. 
>     
>     Using an input field is a nice trick for communicating the fact that the user can type-in (for example, GMail does the same thing with a fake input at the bottom of the emails that looks like a "Reply" field, but that acts as a big button to open the actual email editor), so we can keep the input but change the way we give feedback when typing. 
>     
>     Not sure how to implement the feedback, though. Maybe making the input field flash green when the text was sent and red when it was not? In any case, if it's too much work we can commit this now and change it later. Better to have something done now instead of improving it forever and never releasing :)

So I did both:
1. Add a keyboardstate package that allows to disable the TextField in the plasmoid when the IME is currently not active. It gives a nice effect and a clear feedback to the user. I wanted to add a tooltip when typing is not possible to further inform the user, but then saw that ToolTip was added in 5.7 and gave it up for now.
2. I tried to improve the interpretation of the echoed specialKeys (backspace, delete, arrow keys) to improve local key interaction. Well it's not optimal and maybe we decide to go another way in the future. Still not sure what is best ...

One more thing besides the mentioned changes: Apart from the official Android examples for implementing IMEs I looked at the excellent app from onyxbits (https://github.com/onyxbits/remotekeyboard/) to learn by example for using all these Android APIs. Therefore I wanted to add something like a THANKS-file mentioning the one that invented the wheel for me ;-) Would that be ok, or how do you cope with that usually?

- Holger

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://git.reviewboard.kde.org/r/129727/#review101931
-----------------------------------------------------------

On Jan. 13, 2017, 8:33 p.m., Holger Kaelberer wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://git.reviewboard.kde.org/r/129727/
> -----------------------------------------------------------
> 
> (Updated Jan. 13, 2017, 8:33 p.m.)
> 
> 
> Review request for kdeconnect.
> 
> 
> Bugs: 370919
>     http://bugs.kde.org/show_bug.cgi?id=370919
> 
> 
> Repository: kdeconnect-kde
> 
> 
> Description
> -------
> 
> Allow to inject keypress events to remote peers (most notably Android devices)
> 
> Notes / open issues / possible improvements:
> 
> - For the json-payload I used the syntax of the key-events as sent by mousepad-plugin with the addition of a "sendAck"-flag. If "sendAck" is set to true the remote peer should echo a key-event if it could be handled, thus allowing the local client to find out whether the key was accepted. For performance reasons, it's allowed to send multi-char strings in the "key" property (performs much better if you send a whole string via "echo '...' |  kdeconnect-cli ..." e.g.)
> 
> - kdeconnect-cli: For now takes a string and transforms it into single key-events for visible characters only. In a first implementation I used a kbhit() helper that used termios.h to catch and relay keypresses interactively (including some special-events), which was not optimal. A better approch might be to use linux input-api directly. Would this be an option regarding cross-platform compatibility or can I assume to develop for Linux only? Being a command-line guy, I'd really like to have a fully featured kdeconnect-cli interface ;-)
> 
> - Factor out the Qt::Key-to-internal keymap to some core-helper because it corresponds to the mapping in the mousepad-plugin?
> 
> - The plasmoid is not perfect as it is: A single line containing a non-echoing TextField (i.e. it eats all the KeyPress events), and only ack-ed keypress-packets from the peer device are injected if they contain visible keys. Advantage: the user sees whether his key-presses are accepted by the peer device. Disadvantage: The echoed text does not correspond 1:1 to what is shown on the peer's display, user might be confused when typing without success. I played around with different variations each of which with its proper shortcomings:
> 1. An echoing Textfield for typing: Has the advantage that the user can directly see what he is typing, which makes interaction in the typing field easier, BUT messes up interaction if the Editor on the peer is changed silently and does not notify the user if his keypresses are not handled by the peer.
> 2. A non-echoing TextField for typing PLUS a readonly one for printing visible echoed keys. Disadvantage: same as for the previous one and uses more space on the plasmoid.
> Comments? Ideas?
> 
> 
> Diffs
> -----
> 
>   cli/kdeconnect-cli.cpp 1acb723 
>   interfaces/CMakeLists.txt 19a9cdc 
>   interfaces/dbusinterfaces.h 9848d0c 
>   interfaces/dbusinterfaces.cpp aab3e05 
>   plasmoid/declarativeplugin/kdeconnectdeclarativeplugin.cpp 282f2a9 
>   plasmoid/package/contents/ui/DeviceDelegate.qml e90e021 
>   plasmoid/package/contents/ui/RemoteKeyboard.qml PRE-CREATION 
>   plugins/CMakeLists.txt 9d230ac 
>   plugins/remotekeyboard/CMakeLists.txt PRE-CREATION 
>   plugins/remotekeyboard/README PRE-CREATION 
>   plugins/remotekeyboard/kdeconnect_remotekeyboard.json PRE-CREATION 
>   plugins/remotekeyboard/remotekeyboardplugin.h PRE-CREATION 
>   plugins/remotekeyboard/remotekeyboardplugin.cpp PRE-CREATION 
> 
> Diff: https://git.reviewboard.kde.org/r/129727/diff/
> 
> 
> Testing
> -------
> 
> A lot!
> 
> 
> Thanks,
> 
> Holger Kaelberer
> 
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.kde.org/pipermail/kdeconnect/attachments/20170113/1737426a/attachment-0001.html>