Next Gen CI Will Be Moving to Production Shortly: Upcoming Changes

[Ben Cooksley]

On Sat, May 6, 2017 at 10:15 PM, Elvis Angelaccio
<elvis.angelaccio at kde.org> wrote:
> On sabato 6 maggio 2017 11:37:51 CEST, Ben Cooksley wrote:
>>
>> Hi everyone,
>>
>> This is going to be quite a long email, my apologies in advance for that.
>> If you use the CI system regularly as part of your development flow it
>> is important you read this email in it's entirety as your action will
>> probably be required. If you are aware of a list I have missed in the
>> above, please feel free to forward it there.
>>
>> As some will have been aware of for some time we currently have a
>> problem where changing the system base is an incredibly disruptive
>> activity. We also have issues where builds sometimes fail due to files
>> disappearing mid-build or installations being inconsistent, and
>> excessive numbers of emails are generated. To top this off, everyone
>> has to use the same underlying "base" system for performing builds
>> which sets up conflicts between projects. Oh, and we also only perform
>> CI for Linux.
>>
>> To solve these problems we've been working on a Next Generation CI
>> system which introduces a new concept called 'Products'. In short, a
>> Product is a release unit, such as Frameworks, Plasma, KDE
>> Applications, which groups a series of repositories which are
>> developed and subsequently released together.
>>
>> A preview of this system can be viewed at https://build-sandbox.kde.org/
>>
>> Products as part of their definition are able to define a set of
>> 'Platforms' on which they build. At the moment we have three Platforms
>> available: Ubuntu Xenial Qt 5.7, Windows Qt 5.7 and FreeBSD Qt 5.7.
>> Adding additional Platforms to this mix is fairly easy, as long as the
>> code can be built there. Qt will now be considered as part of the base
>> system, and is something we will no longer build ourselves.
>>
>> Each Product / Platform definition is fully independent. This will
>> allow for different products to build against different versions of Qt
>> among other things for instance.
>>
>> This is the first part that requires your action: we'd like
>> developers, particularly those whose development relies on bleeding
>> edge system software to assist in creating and maintaining (Linux)
>> Platform images. If you're interested in this, please let me know.
>>
>> As part of the shift to the Products paradigm, a number of changes
>> have been made to how projects are built and dependencies managed.
>> This particularly affects dependencies on projects which are outside
>> your Product (Frameworks is outside of Plasma and Applications for
>> instance). These dependencies will now be satisfied through
>> "Dependency Build" jobs, which will be executed once a week. As a
>> result the master version of Frameworks will no longer be available
>> outside Frameworks itself.
>>
>> This change is one of the big ones which massively reduces the effort
>> required to change base systems and thus hugely improves the
>> maintainability of the CI system. It is therefore quite important and
>> necessary, and also isolates the rest of the CI system from breakages
>> lower down in the stack which have happened in the past.
>>
>> This is the second point that requires your attention. If your
>> development process is dependent on using the latest development
>> version of something which is located in another product, then we will
>> need to add that to your Product. If this affects you, please start a
>> new thread (CC'ing sysadmin and kde-core-devel along with your
>> Product's main list) stating which specific repositories you need and
>> providing one to two lines of explaination for each.
>>
>> Note that requests for the entirety of Frameworks won't be accepted -
>> each must be requested on an individual basis with an explanation
>> given for why your development process is dependent on the master
>> version of that Framework.
>>
>> With the change to Products as a core concept for driving the CI
>> system, this does leave Extragear somewhat in a bit of an unusual
>> situation. At the moment we're planning to deal with this by having
>> Extragear be a 'Product' with all of them combined together. If anyone
>> in Extragear has any comments to make on this they're certainly
>> welcome here.
>>
>> Which brings me to the third point of attention. We'll only be adding
>> projects to the Next Gen CI system at their request going forward. For
>> Frameworks, Applications and Plasma this is something which we're
>> essentially assuming we're going to receive from their release
>> managers, so we'll take care of defining the initial Products for
>> those. For Extragear projects, please respond to this thread if you'd
>> like CI coverage (to continue) to be provided to you.
>
>
> What's the role of kde-build-metadata.git in the new CI? If an extragear
> project is already defined there, won't it be automatically built by the new
> CI?

kde-build-metadata continues to be consulted for:

a) Dependencies
b) Branch Group -> Branch associations

If dependencies aren't declared in k-b-m they won't be made available.
Jobs won't become available if a Branch Association hasn't been set
for the Branch Group.

In order for a repository to be considered for job creation however,
it must be a member of one or more Products.
If a repository isn't in any Product, then no CI builds will occur for it.

>
>>
>> Thanks for all who have reached this point - my apologies again for
>> it's length.
>>
>> Note that the Next Gen system isn't finished yet - Notifications are
>> yet to be setup and there are a few other touches left to be made.
>>
>> Regards,
>> Ben Cooksley
>> KDE Sysadmin
>
>
> Cheers,
> Elvis

Cheers,
Ben

>
>>
>>
>