[Kde-pim] 4.11.0 release and tests

[Volker Krause]

On Monday 05 August 2013 21:54:33 Allen Winter wrote:
> On Tuesday, August 06, 2013 01:04:08 AM Albert Astals Cid wrote:
> > @people, please keep both lists on the loop.
> > 
> > One of the things we agreed we wanted for the 4.11 release was no failing
> > tests on jenkins.
> > 
> > We've almost achieved that.
> > 
> > http://build.kde.org/view/KDE%20SC%20stable/?auto_refresh=false
> > 
> > We have only 8 failing tests, all of them in kdepim-*.

Yep, although not entirely surprising considering that's where the majority of 
all our tests seem to be (>1000). That's no excuse of course, just statistics 
;)

Looking at the fail graphs for kdepim*, you can also see that this plan had 
some positive effects :)

> > My question to the kdepim people:
> >  * Why are those tests failing?

Looking at kdepimlibs-stable:

>>> TestSuite.kblog-testmetaweblog	

IIRC that's an online test, probably the remote site changed (happened with a 
few others from kblog as well), deactivate/expected fail IMHO.

>>> TestSuite.Compat-KOrganizer_3.1.ics
>>> TestSuite.Compat-KOrganizer_3.2.ics

IIRC those were fixed with libical 1.0, right Allen?


Looking at kdepim-runtime-stable:

>>> TestSuite.kolabconvertertest

Broken, wasn't adjusted to the kolab resource rewrite I think, should be 
deactivated/expected failed. Christian, can you look into fixing this one 
please?


Looking at kdepim-stable:

>>> TestSuite.ktimezonecomboboxtest
>>> TestSuite.summaryeventtest

those fail with ktimezoned error messages here, probably test setup errors? 
Who could look into that?

>>> TestSuite.messageviewer-rendertest
>>> TestSuite.messagecomposer-messagefactorytest
>>> TestSuite.messagecomposer-maintextjobtest

those are in theory valid tests, but fail for technical reasons (crypto 
setup), non-trivial to fix correctly I think, so temporarily disable/expected 
fail until someone has time to fix the test harness there, I'd say.

> > My question to release-team and kdepim people:
> >  * Should we delay the release until tests are fixed?

IMHO that's not necessary.

> >  * Should we simply QFAIL/remove the tests since it seems they are not
> >  that important?

see above, yes.

> >  * Should we let them fail and still release?
> >
> > Personally i'd like the tests either fixed or QFailed, this way for 4.11.1
> > one can go to jenkins, check if everything is green and have a very high
> > level overview that with some luck maybe nothing regressed.

yes, makes sense

> I agree with you.  We need 0% failing.

Akonadi is at 0% already, can we subscribe kde-pim to Jenkins errors for this 
one already (and add the rest as soon as we hit 0% there for the first time)? 

> 2 of the kdepimlibs fails are from kcalcore (the TestSuite.Compat.KOrganizer
> ones) I'm surprised at this.  Can someone tell me which version of libical
> is installed on the jenkins machines?

IIRC < 1.0, which should explain this.

> We should QFAIL the kblog test.  I can't remember the last time that was
> working.

They use online service and test accounts on these, that's not going to keep 
working long-term without maintenance.

regards,
Volker
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 190 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.kde.org/pipermail/release-team/attachments/20130806/946482a3/attachment.sig>