Next Gen CI Will Be Moving to Production Shortly: Upcoming Changes

[Ben Cooksley]

Hi everyone,

This is going to be quite a long email, my apologies in advance for that.
If you use the CI system regularly as part of your development flow it
is important you read this email in it's entirety as your action will
probably be required. If you are aware of a list I have missed in the
above, please feel free to forward it there.

As some will have been aware of for some time we currently have a
problem where changing the system base is an incredibly disruptive
activity. We also have issues where builds sometimes fail due to files
disappearing mid-build or installations being inconsistent, and
excessive numbers of emails are generated. To top this off, everyone
has to use the same underlying "base" system for performing builds
which sets up conflicts between projects. Oh, and we also only perform
CI for Linux.

To solve these problems we've been working on a Next Generation CI
system which introduces a new concept called 'Products'. In short, a
Product is a release unit, such as Frameworks, Plasma, KDE
Applications, which groups a series of repositories which are
developed and subsequently released together.

A preview of this system can be viewed at https://build-sandbox.kde.org/

Products as part of their definition are able to define a set of
'Platforms' on which they build. At the moment we have three Platforms
available: Ubuntu Xenial Qt 5.7, Windows Qt 5.7 and FreeBSD Qt 5.7.
Adding additional Platforms to this mix is fairly easy, as long as the
code can be built there. Qt will now be considered as part of the base
system, and is something we will no longer build ourselves.

Each Product / Platform definition is fully independent. This will
allow for different products to build against different versions of Qt
among other things for instance.

This is the first part that requires your action: we'd like
developers, particularly those whose development relies on bleeding
edge system software to assist in creating and maintaining (Linux)
Platform images. If you're interested in this, please let me know.

As part of the shift to the Products paradigm, a number of changes
have been made to how projects are built and dependencies managed.
This particularly affects dependencies on projects which are outside
your Product (Frameworks is outside of Plasma and Applications for
instance). These dependencies will now be satisfied through
"Dependency Build" jobs, which will be executed once a week. As a
result the master version of Frameworks will no longer be available
outside Frameworks itself.

This change is one of the big ones which massively reduces the effort
required to change base systems and thus hugely improves the
maintainability of the CI system. It is therefore quite important and
necessary, and also isolates the rest of the CI system from breakages
lower down in the stack which have happened in the past.

This is the second point that requires your attention. If your
development process is dependent on using the latest development
version of something which is located in another product, then we will
need to add that to your Product. If this affects you, please start a
new thread (CC'ing sysadmin and kde-core-devel along with your
Product's main list) stating which specific repositories you need and
providing one to two lines of explaination for each.

Note that requests for the entirety of Frameworks won't be accepted -
each must be requested on an individual basis with an explanation
given for why your development process is dependent on the master
version of that Framework.

With the change to Products as a core concept for driving the CI
system, this does leave Extragear somewhat in a bit of an unusual
situation. At the moment we're planning to deal with this by having
Extragear be a 'Product' with all of them combined together. If anyone
in Extragear has any comments to make on this they're certainly
welcome here.

Which brings me to the third point of attention. We'll only be adding
projects to the Next Gen CI system at their request going forward. For
Frameworks, Applications and Plasma this is something which we're
essentially assuming we're going to receive from their release
managers, so we'll take care of defining the initial Products for
those. For Extragear projects, please respond to this thread if you'd
like CI coverage (to continue) to be provided to you.

Thanks for all who have reached this point - my apologies again for
it's length.

Note that the Next Gen system isn't finished yet - Notifications are
yet to be setup and there are a few other touches left to be made.

Regards,
Ben Cooksley
KDE Sysadmin