How to handle KDE not respecting YOUR distros requirements?

[Martin Graesslin]

On Tuesday, April 12, 2016 12:42:41 PM CEST Richard Brown wrote:
> On 12 April 2016 at 11:04, Martin Graesslin <mgraesslin at kde.org> wrote:
> > On Sunday, April 10, 2016 10:41:20 PM CEST Richard Brown wrote:
> >> Is there anyone from this list going to be taking me up on this offer?
> > 
> > our sysadmins who would have to setup an openQA infrastructure live in
> > Australia and USA/west coast. I doubt they will attend openSUSE conf.
> 
> That's a shame :(
> 
> >> I'm working on the abstract for my proposal to the CFP and I want to
> >> get the balance right between either a general openQA talk or if I
> >> should be putting a KDE spin on it if there is going to be people from
> >> here attending it ;)
> > 
> > I have not yet decided whether I will go to openSUSE conf. In general I'm
> > very interested in openQA and similar technologies. Especially to know
> > whether they provide an additional benefit over our current test
> > infrastructure at build.kde.org. Our biggest areas of quality issues are
> > things like: * what happens when you physically unplug a screen
> > * what happens if new Qt version regressed in multi-screen handling
> > * what happens if the Intel xorg driver regressed [1]
> > 
> > Do you think OpenQA could help us detect these problems automatically? If
> > so, I would be very interested to learn more. If not, we may have to look
> > for other solutions, after all.
> 
> Driver Regressions - Yes. We have the 'GeneralHW' backend which allows
> us to use arbitrary IP KVM VNC devices (eg
> http://www.adder.com/products/adderlink-ipeps ) to look at the real
> VGA output of actual video cards. This is already fully implimented
> and used inside SUSE.

so you have multiple hardware like one server with
* Intel IvyBridge
* Intel SandyBridge
* Intel Haswell
* same matrix for AMD cards
* same matrix for NVIDIA cards
* in the last two cases running all tests with prop and free driver

That would be awesome!

> 
> Physical Unplug - Yes. Using QEMU's device_del function we should be
> hotplug and unplug video cards. This will need a little bit of openQA
> backend work to make viable, but not much.

I rather meant the behavior of real world system - not something emulated. 
Like how does the X server's intel driver behave if one unplugs the output. 
I'm not really interested in the result of virtual devices as that doesn't 
help us: it needs to work on real hardware [1]. Virtual is better than 
nothing, but virtual we already have.

> 
> Multi-screen handling. Yes in Theory . openQA typically looks at only
> one screen at the moment, but there is no reason we couldn't expand
> it.

That would be very useful also for openSUSE. It's a pain point especially as 
there is no way to properly test this with Xorg.

> 
> > Especially the fact that one can hardly auto-test xrandr is a huge problem
> > to us. On the other hand we work a lot on making this testable - our
> > Wayland efforts are completely designed around making this testable. With
> > KWayland having a line test coverage of > 80 %, kscreen having dedicated
> > test infrastructure allowing to feed in configurations. That's all things
> > our current infrastructure can already handle.
> 
> Do your tests actually confirm that it's working from a users
> perspective though? Can I see some examples? I'm curious

Sure, for KWin we have quite a bit of integration testing nowadays which 
verify not just that a piece of code works, but that the actual interaction 
works. An example for such a test is that double clicking on a window 
decoration performs the expected action, see: https://quickgit.kde.org/?
p=kwin.git&a=blob&h=fa000ed65e44b56d0d7e94f7664e0b37f47d04e8&hb=5646313c2c98bf9d064c1be62b5d4166876400e0&f=autotests
%2Fwayland%2Fdecoration_input_test.cpp

Each test starts a full KWin, then creates windows (either Wayland or X11, or 
even by starting external applications). The test infrastructure can verify 
all aspects about the windows (like position, size), can simulate rendering 
and verify the rendering results, can simulate input events, etc. etc.

We have the tech in place for a few months now, so not everything is tested 
yet. But we are constantly extending. All new code, every bug fix now gets a 
test case. My hope is that we can use the technology also to test more 
aspects, like starting a plasmashell to e.g. verify that a menu opened, is 
rendered correctly, etc.

Of course it's only virtual which limits the use cases and why I am looking 
for ways which would allow us to test real hardware conditions. Though of 
course the two biggest pain points - missing xrandr and intel xorg driver - 
don't matter in this setup.

Cheers
Martin

[1] That's why user testing is still important. There we can ask users to 
actually test that with real existing hardware.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 181 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.kde.org/pipermail/distributions/attachments/20160412/704927f7/attachment.sig>