[Kde-pim] 4.11.0 release and tests

[Ben Cooksley]

On Tue, Aug 6, 2013 at 7:47 PM, Volker Krause <vkrause at kde.org> wrote:
> On Monday 05 August 2013 21:54:33 Allen Winter wrote:
>> On Tuesday, August 06, 2013 01:04:08 AM Albert Astals Cid wrote:
>> > @people, please keep both lists on the loop.
>> >
>> > One of the things we agreed we wanted for the 4.11 release was no failing
>> > tests on jenkins.
>> >
>> > We've almost achieved that.
>> >
>> > http://build.kde.org/view/KDE%20SC%20stable/?auto_refresh=false
>> >
>> > We have only 8 failing tests, all of them in kdepim-*.
>
> Yep, although not entirely surprising considering that's where the majority of
> all our tests seem to be (>1000). That's no excuse of course, just statistics
> ;)
>
> Looking at the fail graphs for kdepim*, you can also see that this plan had
> some positive effects :)
>
>> > My question to the kdepim people:
>> >  * Why are those tests failing?
>
> Looking at kdepimlibs-stable:
>
>>>> TestSuite.kblog-testmetaweblog
>
> IIRC that's an online test, probably the remote site changed (happened with a
> few others from kblog as well), deactivate/expected fail IMHO.
>
>>>> TestSuite.Compat-KOrganizer_3.1.ics
>>>> TestSuite.Compat-KOrganizer_3.2.ics
>
> IIRC those were fixed with libical 1.0, right Allen?
>
>
> Looking at kdepim-runtime-stable:
>
>>>> TestSuite.kolabconvertertest
>
> Broken, wasn't adjusted to the kolab resource rewrite I think, should be
> deactivated/expected failed. Christian, can you look into fixing this one
> please?
>
>
> Looking at kdepim-stable:
>
>>>> TestSuite.ktimezonecomboboxtest
>>>> TestSuite.summaryeventtest
>
> those fail with ktimezoned error messages here, probably test setup errors?
> Who could look into that?

That is quite an unstable test according to Jenkins - see the below link.
http://build.kde.org/job/kdepim_master/lastCompletedBuild/testReport/(root)/TestSuite/ktimezonecomboboxtest/history/?

>
>>>> TestSuite.messageviewer-rendertest
>>>> TestSuite.messagecomposer-messagefactorytest
>>>> TestSuite.messagecomposer-maintextjobtest
>
> those are in theory valid tests, but fail for technical reasons (crypto
> setup), non-trivial to fix correctly I think, so temporarily disable/expected
> fail until someone has time to fix the test harness there, I'd say.

messageviewer-rendertest seems to be failing due to changes in the
HTML code itself.
messagecomposer-maintextjobtest appears to be related to charset issues.
messagecomposer-messagefactorytest fails due to header changes from
what I can tell.

You can navigate through the Jenkins interface to get the logs from
the test executions if you wish.

>
>> > My question to release-team and kdepim people:
>> >  * Should we delay the release until tests are fixed?
>
> IMHO that's not necessary.
>
>> >  * Should we simply QFAIL/remove the tests since it seems they are not
>> >  that important?
>
> see above, yes.
>
>> >  * Should we let them fail and still release?
>> >
>> > Personally i'd like the tests either fixed or QFailed, this way for 4.11.1
>> > one can go to jenkins, check if everything is green and have a very high
>> > level overview that with some luck maybe nothing regressed.
>
> yes, makes sense
>
>> I agree with you.  We need 0% failing.
>
> Akonadi is at 0% already, can we subscribe kde-pim to Jenkins errors for this
> one already (and add the rest as soon as we hit 0% there for the first time)?

Both Akonadi master and Akonadi stable have now been configured to
send emails to kde-pim at kde.org when tests begin to fail, or if the
build fails.

>
>> 2 of the kdepimlibs fails are from kcalcore (the TestSuite.Compat.KOrganizer
>> ones) I'm surprised at this.  Can someone tell me which version of libical
>> is installed on the jenkins machines?
>
> IIRC < 1.0, which should explain this.
>
>> We should QFAIL the kblog test.  I can't remember the last time that was
>> working.
>
> They use online service and test accounts on these, that's not going to keep
> working long-term without maintenance.
>
> regards,
> Volker

Thanks,
Ben Cooksley
KDE Sysadmin

>
> _______________________________________________
> KDE PIM mailing list kde-pim at kde.org
> https://mail.kde.org/mailman/listinfo/kde-pim
> KDE PIM home page at http://pim.kde.org/