Blacklisting of PIM from the CI system

[Ben Cooksley]

On Sun, Dec 1, 2019 at 10:17 AM Volker Krause <vkrause at kde.org> wrote:
>
> sigh...
>
> On Saturday, 30 November 2019 19:14:38 CET Ben Cooksley wrote:
> [...]
> > Which is where the problem with PIM comes in - because it currently
> > has many repositories failing to build from source on all platforms
> > those builds are enabled for (including Linux and FreeBSD).
>
> looking at this right now, at least three errors seem transient (ie. a manual
> rebuild "fixed" them), one I can't explain yet is this one:
>
> https://build.kde.org/job/Applications/view/Everything/job/kpimtextedit/job/
> kf5-qt5%20SUSEQt5.12/39/console
>
> This is about using new API in KF 5.65, but the use is guarded by a
> corresponding version check #ifdef. Which KF5 is the CI building against,
> latest release, latest master, or maybe something in between (the latter would
> explain this)?
>

The CI system essentially uses the latest master of Frameworks - with
the slight catch of it the build being up to a week out of date.
The Frameworks build is updated by the "Dependency Build" jobs that
run for each Product/Branch/Platform combination once a week.

The last time the Dependency Build job ran for the Applications
kf5-qt5 SUSEQt5.12 combination was about 2 days ago.
https://build.kde.org/job/Administration/job/Dependency%20Build%20Applications%20kf5-qt5%20SUSEQt5.12/40/console

Unfortunately this run failed due to breakage in 'pimcommon'.
Before it failed, it did get the chance to build Sonnet so that is up
to date as of the date of that build.

> I'll look through the rest as time permits.

Thanks, that would be appreciated.

>
> > Given that the PIM project mailing list is emailed by the CI system,
> > and that the changes in one case were pushed over 2 days ago, there is
> > no excuse for this series of build failures.
>
> I try to actively look at the CI state every 24h, however that sometimes fails
> when I'm on the road or otherwise busy. I do see the email notifications, but

Thanks for doing so.

> given the high number of transient failures (usually due to unfortunately
> timed dependency bumps), they are of limited use for me for raising an alarm
> in cases of actual breakage.

*nod*

With regards to the dependency bumps, the best way to avoid these is
to first bump version numbers, and then increase the dependency
requirements in the projects in a second round of commits, pushed
separately, once the version bump builds have finished. This is from
my understanding how David does it for Frameworks and it works
flawlessly.

This solution has been proposed in the past, but was rejected by the
PIM team who insisted that it was the CI system that should instead
sequence the builds correctly (something which is unscalable to
implement, and from my understanding impossible with Gitlab CI)

>
> > In addition to all of the above, this round of updates was to lay the
> > ground work for adding additional dependencies which are necessary for
> > the builds of Digikam and SubtitleComposer to commence on the CI
> > system for Windows. These failures by PIM have therefore indirectly
> > harmed other KDE projects.
> >
> > As this is not the first time that PIM has caused issues in this
> > manner, I would therefore like to proceed with blacklisting PIM from
> > the CI system. This would include prohibiting other projects from
> > depending on any part of PIM, so those projects that have a required
> > dependency on PIM would also have their builds removed by this.
>
> Of course stuff tends to break more often when you look at 50+ actively
> developed repos compared to a single barely alive one. And yes, I do very well
> understand that this can be frustrating.
>
> > Whilst I would prefer another solution to this, given that it is a
> > recurring issue that makes maintenance of the CI system substantially
> > harder, I see the removal of PIM from the equation as the only
> > reasonable path forward.
> >
> > Should the PIM developers wish to avoid this consequence for their
> > actions, they will need to provide an action plan as to how this will
> > be avoided in the future.
>
> I cannot realistically promise more than what I am already doing on this (as
> outlined above), the combination of me not able to pay enough attention to the
> CI state and things blowing up in multiple repos on the same day is very rare
> though. If that's not enough, others need to help here as well.
>
> > Fixing the current set of failures will not prevent this blacklisting
> > action from being carried out - as a recurring issue it needs a
> > permanent solution.
>
> What do you envision this permanent solution to look like?

It's hard to say.

Given the current problem, which is changes being actively made that
break the build, the ideal solution would be the developers who are
making these breakages to actively keep an eye on their jobs on the CI
system.

To avoid the issue with 'transient' failures due to new functions, etc
being introduced to libraries then being used immediately in other
projects, i'd suggest that the developers introduce a delay (30
minutes should be sufficient) between the pushes to give the system a
chance to build the updated libraries. In the long run it would be
nice to shrink the size of the PIM dependency graph, either through
consolidating repositories or reorganising the code to cut the number
of dependencies, which would reduce the number of times you'd need to
wait.

>
> Regards,
> Volker

Cheers,
Ben