Complex text input in Plasma

[Weng Xuetian]

On Saturday, 8 April 2017 12:43:54 PDT，Martin Gräßlin wrote：
> Am 2017-04-08 21:21, schrieb Weng Xuetian:
> > On Saturday, 8 April 2017 09:46:17 PDT，Martin Gräßlin wrote：
> > 
> >> Am 2017-04-08 17:26, schrieb Weng Xuetian:
> >> > You're wrong about the QT_IM_MODULE stuff. To make application to use
> >> > the
> >> > wayland protocol to type (text-input), the implementation must be done
> >> > with
> >> > QT_IM_MODULE=wayland. I don't mind if it is set to certain application
> >> > but set
> >> > it in general won't work. Also to have real virtual keyboard , you need
> >> > to let
> >> > the input method daemon to provides a virtual keyboard implementation.
> >> 
> >> No you are wrong about that one :-) It might be that it used to be
> >> like
> >> that, but Wayland is the default if no QT_IM_MODULE is specified. See
> >> https://code.qt.io/cgit/qt/qtwayland.git/tree/src/client/qwaylandintegrat
> >> ion .cpp#n142
> > 
> > What I want to say is, if you set QT_IM_MODULE to qtvirtualkeyboard,
> > that app
> > can't talk to im daemon via the wayland protocol. The automatic
> > selection
> > procedure is not really a problem there.
> 
> And that's not the case. Our virtual keyboard is using the Wayland text
> input protocol and does not require setting the QT_IM_MODULE at all.
> 
> Please believe a little bit in what I'm writing. I coded this stuff.
> 
> > Also, running it inside kwin could still be problematic. It'll also
> > need to
> > display UI. From what I heard about, to run a qml window inside kwin
> > could
> > even be a problem.
> 
> I don't know what you have heard but all our UI in KWin is written in
> Qml nowadays.
> 
> > The races you pointed about is not really a problem. Weston has already
> > done
> > it well. Unless you want to reinvent a new protocol, using the current
> > wayland
> > protocol essentially does what I said. And if such a thing would
> > interfere
> > with kwin's rendering procedure, then probably kwin should refactor
> > about
> > that.
> 
> Just because Weston doesn't think of this problem doesn't mean it
> doesn't exist. Weston is not perfect and I found several bugs in the way
> they implemented the protocols they developed and Weston is the
> reference for.
> 
> > And let me give you some number here, a modern input method for Chinese
> > could
> > takes about 30ms to handle one single key (it is THAT complex and we
> > are all
> > limited by the complexity, though I measured it about 5 years ago). You
> > really
> > want to have kwin block on that and wait for such a long time? And just
> > because it is such a slow thing, the IPC latency is not even
> > comparable. If
> > you want good prediction or handwriting, it will be slow. There's no im
> > use
> > neural network on linux right now, but iOS's virtual keyboard is
> > already doing
> > that. Which might be the future, but also mean the im processing time
> > might
> > not slow down.
> 
> I don't want to base any decisions on 5 year old numbers. 30 msec would
> be bad, but if it is with modern hardware just 10 it wouldn't be that
> bad.

30ms is almost the maximum time that I found it would be accepted by the user. 
But it also means that if it is ok, we could put more computation into it to 
achieve better result. Also, I do feel that synchronously wait for input 
method result would be bad. In our customized im module era, we are also 
moving to synchronous wait to key event processing result to asynchronous. As 
you said their exists same problem, but overall it improves the 
responsivieness when something does go wrong (when it's not a bug it could be 
memory or disk io).

> 
> > Also the "input method always on" idea is not saying that every single
> > key
> > will be send to input method, it just mean whenever user need type
> > text, it
> > will be forwarded to input method.
> > 
> > I don't know whether I could persuade you, but please look around the
> > world,
> > iOS, android, sailfish (also use wayland), mac OS , (windows used to
> > run im
> > directly inside application.. but now they also have a service for that
> > ), no
> > one would run a im daemon inside the compositor. Are they all ignore
> > the ipc
> > overhead?
> 
> I'm using sailfish and Android and both keyboards suck bad times.
> Sailfish is really bad, well it's maliit so no surprise there. With both
> systems I see problems in UI positioning because it's not in the
> compositor. Both don't feel perfect especially not in the "every frame
> perfect sense" we are aiming for on Wayland.
> 
But afterall, running fcitx as a library is possible since it is how it is 
implemented. Technically there's no such difficulty prevent us to do so.  And I 
have been using fcitx as a library for testing purpose.  But I guess the story 
is different for ibus, ibus is multi-process and its engine always running in 
different process, and it will always need to pass some ipc to its own engine.

On the bright side, I could bypass the ugly zwp_input_method_v1 and directly 
have zwp_text_input_v1 (or even unimplemented v2), I'd love to see that.

While there's still some hidden problem, the input method upgrade won't be 
easy since it's implmented via shared library (dlclose is dangerous that I 
always disable it via DL_NODELETE), I don't know if it would be bothersome to 
user or not, but luckily it's not something that will receive updates 
everyday.