Blacklisting of PIM from the CI system

[Volker Krause]

On Sunday, 1 December 2019 04:00:19 CET Ben Cooksley wrote:
> On Sun, Dec 1, 2019 at 10:17 AM Volker Krause <vkrause at kde.org> wrote:
> > sigh...
> > 
> > On Saturday, 30 November 2019 19:14:38 CET Ben Cooksley wrote:
> > [...]
> > 
> > > Which is where the problem with PIM comes in - because it currently
> > > has many repositories failing to build from source on all platforms
> > > those builds are enabled for (including Linux and FreeBSD).
> > 
> > looking at this right now, at least three errors seem transient (ie. a
> > manual rebuild "fixed" them), one I can't explain yet is this one:
> > 
> > https://build.kde.org/job/Applications/view/Everything/job/kpimtextedit/jo
> > b/ kf5-qt5%20SUSEQt5.12/39/console
> > 
> > This is about using new API in KF 5.65, but the use is guarded by a
> > corresponding version check #ifdef. Which KF5 is the CI building against,
> > latest release, latest master, or maybe something in between (the latter
> > would explain this)?
> 
> The CI system essentially uses the latest master of Frameworks - with
> the slight catch of it the build being up to a week out of date.
> The Frameworks build is updated by the "Dependency Build" jobs that
> run for each Product/Branch/Platform combination once a week.
> 
> The last time the Dependency Build job ran for the Applications
> kf5-qt5 SUSEQt5.12 combination was about 2 days ago.
> https://build.kde.org/job/Administration/job/Dependency%20Build%20Applicatio
> ns%20kf5-qt5%20SUSEQt5.12/40/console
> 
> Unfortunately this run failed due to breakage in 'pimcommon'.
> Before it failed, it did get the chance to build Sonnet so that is up
> to date as of the date of that build.

I suspect that's where the problem is. The "missing" commit from Sonnet is 
from Nov 26, the dependency build is from Nov 28, yet the kpimtextedit build 
still "sees" the old version.

The same problem also exists with kwidgetaddons and korganizer/knotes.

I looked in the code of those cases, and tested this locally. This builds 
against 5.64 and latest 5.65, when I switch frameworks to something in between 
I can reproduce the issues we are seeing on the CI here too. Not sure how to 
fix that in the code, I can't version-check for this...

> > > Given that the PIM project mailing list is emailed by the CI system,
> > > and that the changes in one case were pushed over 2 days ago, there is
> > > no excuse for this series of build failures.
> > 
> > I try to actively look at the CI state every 24h, however that sometimes
> > fails when I'm on the road or otherwise busy. I do see the email
> > notifications, but
> Thanks for doing so.
> 
> > given the high number of transient failures (usually due to unfortunately
> > timed dependency bumps), they are of limited use for me for raising an
> > alarm in cases of actual breakage.
> 
> *nod*
> 
> With regards to the dependency bumps, the best way to avoid these is
> to first bump version numbers, and then increase the dependency
> requirements in the projects in a second round of commits, pushed
> separately, once the version bump builds have finished. This is from
> my understanding how David does it for Frameworks and it works
> flawlessly.
> 
> This solution has been proposed in the past, but was rejected by the
> PIM team who insisted that it was the CI system that should instead
> sequence the builds correctly (something which is unscalable to
> implement, and from my understanding impossible with Gitlab CI)

I agree with that solution, and it's how I try to do dependency bumps when 
necessary.

(I'll reply to the ideas how to improve the situation going forward 
separately/later, I'm in a rush now, sorry).

Regards,
Volker

> > > In addition to all of the above, this round of updates was to lay the
> > > ground work for adding additional dependencies which are necessary for
> > > the builds of Digikam and SubtitleComposer to commence on the CI
> > > system for Windows. These failures by PIM have therefore indirectly
> > > harmed other KDE projects.
> > > 
> > > As this is not the first time that PIM has caused issues in this
> > > manner, I would therefore like to proceed with blacklisting PIM from
> > > the CI system. This would include prohibiting other projects from
> > > depending on any part of PIM, so those projects that have a required
> > > dependency on PIM would also have their builds removed by this.
> > 
> > Of course stuff tends to break more often when you look at 50+ actively
> > developed repos compared to a single barely alive one. And yes, I do very
> > well understand that this can be frustrating.
> > 
> > > Whilst I would prefer another solution to this, given that it is a
> > > recurring issue that makes maintenance of the CI system substantially
> > > harder, I see the removal of PIM from the equation as the only
> > > reasonable path forward.
> > > 
> > > Should the PIM developers wish to avoid this consequence for their
> > > actions, they will need to provide an action plan as to how this will
> > > be avoided in the future.
> > 
> > I cannot realistically promise more than what I am already doing on this
> > (as outlined above), the combination of me not able to pay enough
> > attention to the CI state and things blowing up in multiple repos on the
> > same day is very rare though. If that's not enough, others need to help
> > here as well.
> > 
> > > Fixing the current set of failures will not prevent this blacklisting
> > > action from being carried out - as a recurring issue it needs a
> > > permanent solution.
> > 
> > What do you envision this permanent solution to look like?
> 
> It's hard to say.
> 
> Given the current problem, which is changes being actively made that
> break the build, the ideal solution would be the developers who are
> making these breakages to actively keep an eye on their jobs on the CI
> system.
> 
> To avoid the issue with 'transient' failures due to new functions, etc
> being introduced to libraries then being used immediately in other
> projects, i'd suggest that the developers introduce a delay (30
> minutes should be sufficient) between the pushes to give the system a
> chance to build the updated libraries. In the long run it would be
> nice to shrink the size of the PIM dependency graph, either through
> consolidating repositories or reorganising the code to cut the number
> of dependencies, which would reduce the number of times you'd need to
> wait.
> 
> > Regards,
> > Volker
> 
> Cheers,
> Ben

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 195 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.kde.org/pipermail/kde-core-devel/attachments/20191201/25693acb/attachment.sig>