Scene redesign

[Vlad Zahorodnii]

Hi,

As you all know, KWin has several serious rendering issues. One of them
is the texture bleeding, and the other is related to sub-surfaces. Let's
discuss each problem first and then move to the proposed solution.

On X11, we name the frame window pixmap rather than the client window
pixmap. However, a texture atlas is used to store all individual parts
of the server-side decorations (left, top, right, and bottom). The
server-side decoration is rendered into the frame window only when
compositing is turned off. Given that the client contents is inside a
larger texture with nothing around it and each sprite in the decoration
atlas is not padded, GL_LINEAR filter may introduce a gap between the
decoration and client contents. This is the texture bleeding issue. The
most easiest way to notice it is to drag a wobbly window over a
contrasting background.

On Wayland, no window quads are generated for sub-surfaces. This
basically means that effects like Magic Lamp and Wobbly Windows won't
work as expected for clients that utilize sub-surfaces, e.g. video
players. Another problem is that sub-surfaces cannot be clipped since
they don't have quads. See https://bugs.kde.org/387313

Some of those issues were discussed in
https://phabricator.kde.org/T10530. In general, I don't see how we can
fix texture bleeding and introduce window quads for sub-surfaces without
breaking API. Anyway,
let me go quickly through the proposed solution.

Sub-surfaces form a tree, thus our scene data structures must reflect
that. Though, we could go a bit further and include the server-side
drop-shadow as well the server-side decoration in that tree, e.g.

    ShadowNode
    └── DecorationNode
        └── SurfaceNode (A)
            ├── SurfaceNode (B)
            │   └── SurfaceNode (C)
            └── SurfaceNode (D)

- ShadowNode represents the server-side drop-shadow
- DecorationNode represents the server-side decoration
- SurfaceNode (A) represents the main surface (or pixmap)
- SurfaceNode (B) (C) (D) represent sub-surfaces

Neither ShadowNode nor DecorationNode nor SurfaceNode solve the texture
bleeding problem. Their purpose is to model surface hierarchy and act as
the source of window quads that can be used **internally** by Scene.

In order to fix the texture bleeding, we need to render the client into
an offscreen texture, after that map the offscreen texture on the screen
with all transformations applied. However, we can't implement that since
WindowQuadContents, WindowQuadDecoration, and WindowQuadShadow will have
texture coordinates that are not suitable for the offscreen texture. One
could say that we could adjust the texture coordinates late in the
rendering process, however it is not possible due to the way we render
server-side drop-shadows. Also, adjusting texture coordinates will make
the code much more difficult.

Which brings us to sub-surface window quads. I think the best option is
to not introduce them to libkwineffects. In fact, it will be better not
to expose any window quad types to effects.

Q: What does it mean for us, KWin developers?
A: Two things. First, we'll be able to add proper support for
sub-surfaces that won't suck (I hope). Second, we will have some space
for potential future scene redesigns that won't require changes in third
party effects.

Q: What does it mean for third-party binary effects?
A: Yeah... Effects like shapecorners will stop working.

So, the proposed rendering algorithm looks something like this

1. Generate a WindowQuad suitable for the offscreen texture and pass it
to WindowPrePaintData
2. Call prePaintWindow method. If an effect transforms the window quad
in **any** way, it must set PAINT_WINDOW_TRANSFORMED
3. Call paintWindow method
4. When performPaintWindow() of the scene window has been reached, two
things might happen
4.1. (the window is transformed) Render the window into an offscreen
texture, then map the offscreen texture on the screen. The scene could
check whether the client has been damaged in order to skip the first
step and move straight to the second step, i.e. mapping the offscreen
texture on the screen
4.2. (the window is not transformed) Render the window onto the screen
with optimizations and all that good stuff

Redesign of Scene can be accomplished in two steps
- First, we need to introduce the offscreen rendering mechanism in the
OpenGL scene (BREAKING CHANGE)
- Second, refactor scene.cpp and introduce ShadowNode, SurfaceNode, and
DecorationNode classes (perhaps, split/re-brand WindowPixmap class)

Unfortunately, my proposed solution breaks API compatibility. However,
as I said a bit earlier in this email, I don't see any other way. If you
know a way to do that, I'm all ears. Thoughts? Objections? Suggestions?

Cheers,
Vlad