Plasma 5.7 Test Days scheduled; report from first prep meeting

[Eike Hein]

Hi,

for Plasma 5.7, the Plasma community is going to try something new
to find out its impact on release quality: We're going to run a two-
day test event on IRC, asking users to run through some test cases
we author and tell us the results.

Below are the notes from our very first prep meeting for this event,
as well as the full IRC logs. Many of you have greater experience
with running these kinds of events for your distros (in fact that's
what we've been inspired by) so we'd love to hear your feedback.

We're going to do weekly prep meetings in #plasma every Tuesday at
12:00 UTC, refining the plan. The Test Days themselves are presently
scheduled for June 19th and 20th now, as ballparked earlier during
the Plasma 5.7 Kickoff/scheduling meeting.

Here goes those notes:

attendees: eike, bhushan, jens, riddell, notmart, kbroulik
chair: eike

intro (excerpt, see full log):

[19:02] <Sho_> ok, so we want to hold a Plasma Bug Day on June 19th or 
20th (or both), with developer and user participation, both to find out 
what sort of impact we can have on quality with that sort of thing, and 
also to send a message that we do care about Plasma 5 being reliable at 
this point
[19:02] <jensreu> Also its about trying to find users who could see 
themselves being part.
[19:02] <Sho_> the idea will be to give the impending 5.7 release a good 
run-through and find any regressions/problems that need to be addressed 
for the release
[19:03] <jensreu> So for me the relevant bit is that users who have 
until now not helped out, get a good report with KDE people in general 
and may feel like helping out again
[19:03] <Sho_> that means we should discuss (a) prereqs, (b) test day 
content, (c) assign some tasks in prep, (d) messaging/promo and (e) nail 
down a date

(a) prereqs
- we will need ISO images to base tests on
-- jens will talk to kaos, opensuse, fedora; riddell (neon) was also 
listening in
-- rotating isos in future test days can be good for distro relations, 
it's all one team

(b) test day content
- we want testcases a la
https://fedoraproject.org/wiki/Test_Results:Fedora_23_Final_RC10_Desktop 
(final table)
- avoids everyone bringing their pet bug, instead they get asked to 
accomplish a task and log the results, and find new problems along the way
- unclear how specific or coarse test cases need to be (example: "add a 
widget and position it to your liking"), experience will tell
- VDG will produce some example test case content for us to discuss and 
mull more at next week's meeting
- sample test case areas: widget management, panel management, common 
customization settings, device control (network, audio), launching apps, 
window management, ..

(c) messaging/promo
- banners, stickers etc for test days (webbanner - talk to eV about 
sticker printing)
- blog posts
- "how to file a bug report" infographic
- promo launch deadline is first week of june (three weeks before test 
day), materials for blogs need to be ready by then

(d) workflow during test day
- direct to bugzilla vs logging results on wiki, then devs generating 
their own reports from it
- pro bugzilla: educating users to use the real thing
- pro wiki: avoids torrent of issues on bugzilla; devs have to do triage 
either way
- no final decision, leaning towards wiki, but may need more feedback

(e) dates
- weekly prep meetings; next meeting tuesday 12:00 UTC in #plasma again
- our test day will be Test Days: June 19th (Sunday) + June 20th 
(Monday) to catch users off-work and devs at-work
- two stages: Sunday might be focused on collection, giving devs to roll 
in on Monday content to work on
- means it will collide with plasma monday hangout, though ...


Cheers,
Eike
-------------- next part --------------
[19:02] <Sho_> ok, so we want to hold a Plasma Bug Day on June 19th or 20th (or both), with developer and user participation, both to find out what sort of impact we can have on quality with that sort of thing, and also to send a message that we do care about Plasma 5 being reliable at this point
[19:02] <jensreu> +1
[19:02] <jensreu> Also its about trying to find users who could see themselves being part.
[19:02] <Sho_> the idea will be to give the impending 5.7 release a good run-through and find any regressions/problems that need to be addressed for the release
[19:03] <jensreu> So for me the relevant bit is that users who have until now not helped out, get a good report with KDE people in general and may feel like helping out again
[19:03] <Sho_> that means we should discuss (a) prereqs, (b) test day content, (c) assign some tasks in prep, (d) messaging/promo and (e) nail down a date
[19:03] <kbroulik> yay kubuntu, colleague just ame to me "I dont get any window deorations anymore", turns out for some reason kde-window-manager pakage was not installed. wat.
[19:03] <Sho_> (a) prereqs is for me mostly about having an ISO image available we can base tests on
[19:04] <jensreu> Sho_: yeah good points - Neon/Argon/Kaos? Or would that just be too shattered and the bugs may be too distrodependant?
[19:04] <Sho_> so I think we need to talk to Riddell/sitter about whether Neon can deliver us a fresh image by our target date
[19:04] <Sho_> jensreu: I think as long as it's fairly-vanilla 5.7 git code it'd do fine
[19:05] <Sho_> we can make sure our test cases are more plasma-focused and avoid distro integration stuff, if possible
[19:05] <bshah> okay
[19:05] <-- rahulch (~rahul at 116.193.132.40) has left this server (Quit: Konversation terminated!).
[19:05] <jensreu> Sho_: should I talk to Anke (Ankke? don't know her name) about it? She may be interested helping out there
[19:05] <sKreamer> ::plasma5:: Plasma 5 :: Re: plasma 5 on centos 7 @ https://forum.kde.org/viewtopic.php?f=289&t=124990&p=355187#p355187 (by Alloysious)
[19:05] <jensreu> (Since Kaos is vanilla and close to upstream)
[19:05] <Sho_> jensreu: sounds good
[19:05] <Riddell> Sho_: yes we should
[19:05] <bshah> however, it is also good to test plasma in different distros...
[19:05] <Sho_> jensreu: it might actually also be really great for our distro relations if we use different isos for different test days in the future
[19:05] <jensreu> ok
[19:06] <Sho_> jensreu: we can make distros feel part of the team by using their work in our test days
[19:06] <jensreu> Sho_: I could try to get some Opensuse people in on it as well as Fedora's people
[19:06] <jensreu> Ok wait I think we need notes
[19:06] <Sho_> i was going to collect/keep notes and send them out later, but we can also open a notepad alongside, sure :)
[19:07] <jensreu> Errr I have had the weird two days so my brain is shot. Can anyone else start it (cant even remember the addy for it :D )
[19:07] <-- mgraesslin (~martin at kde/mgraesslin) has left this server (Ping timeout: 250 seconds).
[19:08] <Sho_> on it
[19:08] <bshah> https://notes.kde.org/p/plasma-bugday
[19:08] <bshah> err
[19:08] <Sho_> https://notes.kde.org/p/plasma-57-testday
[19:09] <bshah> right: /me destroys his notes
[19:09] <jensreu> Okok :)
[19:09] <Sho_> heh, sorry & thanks
[19:09] <jensreu> thanks
[19:09] <Sho_> ok, so we got some leads on isos
[19:09] <Sho_> now as for test day content, i think that's much of the real meat
[19:09] <jensreu> Yes
[19:09] <Sho_> one of the inspirations i have there is this: https://fedoraproject.org/wiki/Test_Results:Fedora_23_Final_RC10_Desktop
[19:10] <Sho_> in particular, scroll down to the final table on that page, where QA test cases are linked in subpages
[19:10] * notmart arrives and reads backlog
[19:10] <Sho_> what fedora does there is run testing days on irc, and ask users to run those testcases on their RC images, and then log the results in the wiki
[19:11] <bshah> Sho_: those testcases are decided by whom?
[19:11] <Sho_> i think something like this would help with producing useful output and user engagement
[19:11] <jensreu> That sounds like an awesome method. Also (sry for being the political one) it would be good to talk to them about how they do them and how to implement it on Plasma - just to make them feel a part too
[19:11] <Sho_> because i think what we need to avoid is everybody showing up and just bringing along their pet bug
[19:11] <jensreu> +1000 Sho_
[19:11] <Sho_> instead we need to ask them to do something, and find new problems along those paths
[19:11] <Sho_> bshah: they're developed be fedora QA, I guess
[19:12] <Sho_> soooo this of course means work for us to emulate
[19:12] --> apol (~apol at 62.57.48.198.dyn.user.ono.com) has joined this channel.
[19:12] <jensreu> So how do we set the tests - can there be low hanging fruit tests (like design tests along the lines of "Change colour theme, look at icon, can you read the text?"
[19:12] <Sho_> jensreu: definitely
[19:13] <bshah> We right now don't have such QA tasks defined currently..
[19:13] <Sho_> jensreu: i was thinking about cases like "add a widget and arrange it to your liking", that kind of thing
[19:13] <jensreu> (because that may involve people who feel nervous about technical stuff and still wants to help out)
[19:13] <jensreu> Sounds perfect Sho_
[19:13] <Sho_> it's also a question of how specific and/or coarse the tests need to be to produce useful results
[19:13] <Sho_> but i think that's something we will learn from experience
[19:13] <bshah> possibly we should crowd-source ideas for this till up to test days
[19:13] <bshah> (i.e start wiki(?))
[19:13] <Sho_> ok, so what i propose is that one of us sits down and writes some sample test cases, and then at our next prep meeting we look at them and use them as a launching point for writing more cases, or fixing those
[19:14] <Sho_> because if we go with "ok, everyhone start writing testcases" we don't nail down a format and possibly do duplicate work
[19:14] <Sho_> so i think it should be one person for one-two weeks, and then bring in the team again and go wider
[19:14] <jensreu> Ok I would love to get the VDG to collect a set of tests for usability/visuals
[19:14] <Sho_> oh yeah that could work too
[19:14] <Sho_> do you think the vdg would want to write those sample cases for us?
[19:15] <jensreu> Yeah sure... we may need technical help though
[19:15] <Sho_> we can give that of course
[19:16] <Sho_> jensreu: what timeframe do you think would be needed to produce just some sample content we can discuss at a next meeting?
[19:16] <jensreu> Plus I'd really would love for there to be some nice infographic things with "how do you fill out a bugreport" but thats later in the meeting I supose
[19:16] <Sho_> i.e. would we meet next tuesday, or two weeks from now?
[19:16] <jensreu> Next tuesday would work for me
[19:17] <bshah> +1 for weekly meeting if possible
[19:17] <jensreu> I mean the only fear I have is that it will be very much "random tech tests" with no real relevance to what we have
[19:17] <jensreu> +1 for weekly
[19:17] <Sho_> k
[19:17] <Sho_> works for me +1
[19:17] <Sho_> logging that jens will delegate sample test cases to the vdg
[19:17] <jensreu> So that what you guys get back is essentially the VDG doing a scattershot attempt at "techy use cases" to test against
[19:18] <jensreu> +1
[19:18] <Sho_> next is messaging/promo
[19:18] <jensreu> (about the logging me)
[19:18] <jensreu> Yeah so we have some visuals already available - the Konqi Bughunter is a good one
[19:18] <Sho_> i think we want one of those cute "join our test day" banners and us writing blog posts to invite people
[19:18] --> Jucato (~Juan at kde/developer/jucato) has joined this channel.
[19:18] <Sho_> yep
[19:18] <jensreu> And we could make stickers - banners etc out of that
[19:18] <Sho_> sounds good
[19:18] <Sho_> how far in advance should we rev up promo for it?
[19:19] <Sho_> start early june?
[19:19] <jensreu> Plus, some easy infographic for "how the hell do you fill out a bugreport" would be GREAT
[19:19] <jensreu> Like something where we sit down and talk about what devs actually NEED and how to explain it to someone who has never filled in a bug - but with pretty visuals and images
[19:19] <Sho_> yep i agree
[19:20] <Sho_> that would be wonderful
[19:20] <jensreu> Something we can save - and show others after the bughunt is over. Since so many simply dont know
[19:20] <jensreu> (I would LOVE to make a stab at that infographic personally btw)
[19:20] <bshah> topic in #kde-bugs or some similar channel have nice page on that
[19:20] <Sho_> though after we get our sample test cases we need to decide whether we should output into a wiki ala fedora or go bugzilla immediately
[19:20] <Sho_> that workflow stuff is going to be sort of a separate topic in and of itself
[19:21] <Sho_> we need to bridge to bugzilla eventually for devs, but we might do that work instead of the users
[19:21] <jensreu> That I think is best up to you guys (devs) to decide - since that will affect you most
[19:21] <jensreu> Sho_: can it be automated in a nice way?
[19:21] <Sho_> no i think this needs human hands
[19:21] <jensreu> Like fill in this web form -> sends it to bugzilla after someone checks that it is a proper bug
[19:22] <Sho_> with my dev hat on i think whether i do triage in bugzilla, or read a wiki and generate bugzilla from it is not a big diff, but avoids bugzilla becoming messy
[19:22] <jensreu> Ah ok
[19:22] <Sho_> and test days will be a deluge of sorts
[19:22] <Sho_> so using the wiki sounds better to me
[19:22] <Sho_> since it's easier to ignore and forget a wiki page
[19:22] <jensreu> Ok I am all for...
[19:22] <jensreu> :)
[19:22] <Sho_> k
[19:22] <Sho_> ok, so to talk promo timeline
[19:22] <Sho_> do we try to get material ready for first week of june and start blogging etc then?
[19:22] <jensreu> Only one thing though: since then the people taking part will not need to learn bugzilla that may be bad for later involvement. Like they may feel they want to use the form instead
[19:23] <Sho_> jensreu: true
[19:23] <soee> hey guys, second user had similar issue: https://bugs.kde.org/show_bug.cgi?id=360777
[19:23] <bugbot> KDE bug 360777 in kdeplasma-addons (General) "Due to faulty configuration desktop loading time is ~20 seconds longer" [Normal,Unconfirmed: ]
[19:23] <Sho_> soee: sorry, meeting right now :)
[19:23] <soee> ah sorry :)
[19:23] <jensreu> Yeah the sooner the better tbh. The issue with Promo is that we (KDE) often wait until the very last second with it. And since it can happen independently we might as well start it ASAP
[19:24] <jensreu> Sry added notes Sho_ is that ok?
[19:24] <Sho_> jensreu: of course!
[19:24] <Sho_> jensreu: I think we shouldn't start promo too soon though, to avoid it falling out of user's minds
[19:25] <Sho_> three-four weeks before sounds good to me, but i don't have much promo experience
[19:25] <jensreu> No I ment just the work with promo things (like stickers, webbanners infographic etc) just so we will have it ready for when we want to use
[19:25] <jensreu> it
[19:25] <Sho_> ah yeah we need to work on the materials earlier
[19:25] <Sho_> ok, so targeting first week of june as promo launch deadline?
[19:25] <jensreu> Perfect
[19:25] <Sho_> awesome
[19:26] <bshah> but, I think we should reach to distributions early
[19:26] <bshah> at least over distributions outreach list
[19:26] <Sho_> makes sense
[19:26] <bshah> or some other form of communication
[19:26] <Sho_> i can actually CC our meeting notes to the distro list today, and ask them for early feedback
[19:26] <Sho_> sounds good?
[19:26] <jensreu> Yeah I was hoping Thomas could do Opensuse since he has a good report with them right now
[19:27] <Sho_> or jens, do you want an opportunity to tlak to people first?
[19:27] <jensreu> Oh yeah +1 Sho_
[19:27] <Sho_> ah ok, list CC it is then
[19:27] <jensreu> No might as well do every channel available
[19:27] <Sho_> yep
[19:27] <bshah> Sho_: +1 or maybe prepare seperate draft with proper email?
[19:27] <Sho_> ok, final topic is the actual test day date then
[19:27] <jensreu> I mean the more we talk about it with them the better
[19:27] <Sho_> june 19th is sunday, which means people aren't at work
[19:27] <Sho_> june 20th is monday, which means we don't need to work a sunday
[19:27] <Sho_> ;)
[19:27] <Sho_> i think 19th makes more sense in terms of the goal
[19:27] <bshah> wasn't it july?
[19:27] <bshah> o_O
[19:28] <Sho_> oh was it? /me looks
[19:28] <bshah> no ignore me
[19:28] <Sho_> no it's june yep
[19:28] <Sho_> so, do we do 19th? or do we need two days ...? thoughts?
[19:28] <jensreu> Okokok ... time is collapsing in on us
[19:28] <jensreu> Two days might be for the best
[19:29] <notmart> uh, won't be here 19th june, oh well will see the results :)
[19:29] <bshah> June 19th is best for $users and June 20th is best for $devs
[19:29] <bshah> so.. lets have two days
[19:29] <Sho_> ok, let's do two
[19:29] <jensreu> Since we want users around the globe and it will give us an extra day if there is too much stuff to do
[19:29] <Sho_> noting: it will collide with plasma monday hangout then
[19:29] <bshah> or maybe lets have it in two stages : Sunday for reporting and Monday for fixing or at least triaging
[19:30] <Sho_> but on the other hand it means people are used to showing up on mondays ;)
[19:30] <jensreu> I just realized I MAY be on an airplane then... (need to check with my secretary aka "my husband")
[19:30] <bshah> jensreu: Sho_: how's idea of two stage bugdays?
[19:31] <Sho_> bshah: sounds pretty good actually
[19:31] <Sho_> kind of producing bug content for devs to attend to on monday
[19:31] <jensreu> I think it makes sense (sry just trying to find out when my trip is)
[19:31] <Sho_> :)
[19:32] <jensreu> Oh one thing: As for test cases - are there any we should focus on specifically? I mean where we KNOW ther ehave been issues in the past?
[19:33] <Sho_> jensreu: widget management, panel management, common customization settings, device control (network, audio), launching apps, window management, ..
[19:33] *** ochurlaud is now known as ochurlaud--away.
[19:33] <jensreu> Sho_: ok :)
[19:34] --> ivan||home (~ivan at cable-178-148-107-202.dynamic.sbb.rs) has joined this channel.
[19:34] <jensreu> Wth Sho_*I* was going to write the examples in the notes ;)
[19:34] <Sho_> haha, sorry
[19:34] <Sho_> feel free to augment/extend :)
[19:34] <Sho_> i thiiiink that means we got it all for this week's first meeting
[19:34] <Sho_> anybody got more?
[19:35] <jensreu> Nothing from me
[19:35] <jensreu> bshah: ?
[19:35] <bshah> can we make wednesday, meeting day?
[19:35] <bshah> given we have another $meeting on tuesday
[19:35] <jensreu> As long as its around this time I'm fine for wednesday
[19:35] <jensreu> (I have another meeting wednesdays)
[19:35] <Sho_> hmmm tbh i'd actually kind of prefer compressing the meetings so i have uninterrupted coding time on wednesday
[19:36] <Sho_> but if it's a huge inconvenience for you i can do wednesday
[19:36] <bshah> okay... tuesday then it is..
[19:36] <bshah> no, its fine
[19:36] <Sho_> ok, thanks :)
[19:36] <Sho_> alright, so
[19:36] <Sho_> i will keep the notes editing window open until :40
[19:36] <Sho_> and then send out notes and log to plasma and distro lists
[19:37] <jensreu> Awesome
[19:37] <Sho_> thanks all!
[19:37] <bshah> thanks for hosting
[19:37] <jensreu> thanks for hosting
[19:37] <Sho_> :)