Complex text input in Plasma

[Weng Xuetian]

On Saturday, 8 April 2017 09:46:17 PDT，Martin Gräßlin wrote：
> Am 2017-04-08 17:26, schrieb Weng Xuetian:
> > You're wrong about the QT_IM_MODULE stuff. To make application to use
> > the
> > wayland protocol to type (text-input), the implementation must be done
> > with
> > QT_IM_MODULE=wayland. I don't mind if it is set to certain application
> > but set
> > it in general won't work. Also to have real virtual keyboard , you need
> > to let
> > the input method daemon to provides a virtual keyboard implementation.
> 
> No you are wrong about that one :-) It might be that it used to be like
> that, but Wayland is the default if no QT_IM_MODULE is specified. See
> https://code.qt.io/cgit/qt/qtwayland.git/tree/src/client/qwaylandintegration
> .cpp#n142

What I want to say is, if you set QT_IM_MODULE to qtvirtualkeyboard, that app  
can't talk to im daemon via the wayland protocol. The automatic selection 
procedure is not really a problem there.

> > And also, merging more and more daemon into kwin is not always good
> > even from
> > security point of view. The problem is, once it got merged, the whole
> > memory
> > space is being exposed. Which means, if there's a single piece of code
> > is
> > vulnerable, it will affect the whole compositor. We are not perfect
> > people, and
> > that's why put more code together will make it more vulnerable to
> > attacker. If
> > you consider that, your prevention of ptrace on kwin becomes nothing
> > and so
> > does your effort to make kwin not loading some random plugin (prevent
> > ld_preload and qt_plugins_path?).
> 
> The security of the system breaks with the weakest link. Whether the IM
> daemon is insecure by running standalone or inside KWin isn't a
> difference.
> 
> > So, my proposal to this will be:
> > im daemon <-> kwin <-> application, and the communication is done with
> > some
> > wayland protocol.
> > 
> > and imho that would be best option to all people. Here's the reason
> > 
> > Pros
> > - less code to run within kwin process
> > - less change needed to existing im daemon (we just need an extra
> > frontend)
> > and kwin
> > - no hard dependency
> > 
> > Cons:
> > more ipc latency, but this only happens if user need to type.
> 
> The IPC latency is a huge problem here. If we want to have a good
> experience KWin needs to decide whether to send a key event to the IM
> daemon or through wl_keyboard interface to the application. This means
> possibly a roundtrip in the event handling code. This code is currently
> the most crucial part of KWin. It opens a can of worms if we go IPC
> there and given your suggestion it needs IPC.
> 
> And even if we handle it without roundtrip we run into timing issues.
> Consider:
>   * KWin gets key event
>   * KWin decides it goes to IM for window foo
>   * IM does something with it
>   * window foo closes
>   * IM sends the composed text for window foo to KWin
> 
> Now what? In case we would eliminate the IPC by making KWin link the IM
> module it becomes:
> 
>   * KWin gets key event
>   * KWin decides it goes to IM for window foo
>   * IM returns the composed text for window foo
>   * KWin forwards the composed text to window foo
>   * window foo closes
> 
> So from KWin perspective everything becomes cleaner if there is no IPC
> involved towards the IM daemon. In that case KWin needs to adjust way
> less as we don't need a dedicated protocol.
> 
> I understand your concerns with linking stuff and loading plugins but I
> don't share them. Yes I also see this as a problem but I think it's less
> of a problem than IPC in this crucial area.
Then why IPC exists in the first place? The whole wayland stuff IS IPC. Why 
there ARE processes?..

Also, running it inside kwin could still be problematic. It'll also need to 
display UI. From what I heard about, to run a qml window inside kwin could 
even be a problem.

The races you pointed about is not really a problem. Weston has already done 
it well. Unless you want to reinvent a new protocol, using the current wayland 
protocol essentially does what I said. And if such a thing would interfere 
with kwin's rendering procedure, then probably kwin should refactor about 
that.

And let me give you some number here, a modern input method for Chinese could 
takes about 30ms to handle one single key (it is THAT complex and we are all 
limited by the complexity, though I measured it about 5 years ago). You really 
want to have kwin block on that and wait for such a long time? And just 
because it is such a slow thing, the IPC latency is not even comparable. If 
you want good prediction or handwriting, it will be slow. There's no im use 
neural network on linux right now, but iOS's virtual keyboard is already doing 
that. Which might be the future, but also mean the im processing time might 
not slow down.

Also the "input method always on" idea is not saying that every single key 
will be send to input method, it just mean whenever user need type text, it 
will be forwarded to input method. 

I don't know whether I could persuade you, but please look around the world, 
iOS, android, sailfish (also use wayland), mac OS , (windows used to run im 
directly inside application.. but now they also have a service for that ), no 
one would run a im daemon inside the compositor. Are they all ignore the ipc 
overhead?

> 
> Cheers
> Martin