CI talk (Was: re: Manner in which kde-gtk-config development is conducted)

[Ben Cooksley]

On Sun, Mar 22, 2020 at 12:00 PM Johan Ouwerkerk <jm.ouwerkerk at gmail.com> wrote:
>
> On Sat, Mar 21, 2020 at 10:27 PM Ben Cooksley <bcooksley at kde.org> wrote:
> >
> > On Sun, Mar 22, 2020 at 3:27 AM Johan Ouwerkerk <jm.ouwerkerk at gmail.com> wrote:
> > >
> > > On Sat, Mar 21, 2020 at 1:32 AM Ben Cooksley <bcooksley at kde.org> wrote:
> > > >
> > > > Comments welcome. Please note that simply fixing the dependency
> > > > breakage in this case is not enough to resolve this - there are
> > > > underlying issues which need to be addressed here.
> > > >
> > > > Regards,
> > > > Ben Cooksley
> > > > KDE Sysadmin
> > >
> > > I cannot comment as to whether or not this is a pattern of behaviour
> > > or just a few isolated instances. From a technical perspective I feel
> > > there are two (additional) underlying issues worth addressing here:
> > >
> > >  1. This could be prevented for the most part by having CI run before,
> > > and not after the fact. I.e. prior to merging code.
> >
> > This will happen once we are on Gitlab.
> >
> > >  2. Different projects have different CI needs, and it would help if a
> > > project could safely manage their CI environment "on their own" as
> > > much as possible. The current system requires a lot of daunting
> > > (possibly otherwise unnecessary) complexity purely to manage the fact
> > > that a builder image will be used not just for one project but for
> > > perhaps the whole of KDE.
> >
> > The global builder image doesn't cause too many issues fortunately and
> > isn't the problem here. Our current setup actually has the flexibility
> > to use different images - we're just constrained in terms of setting
> > this up by Jenkins as there is a bunch of tooling needed to provision
> > jobs.
> >
> > A good proportion of the current "complexity" is due to our use of
> > Jenkins. The remainder is due to needing to provide the most recent
> > version of a given KDE project as a dependency, along with the
> > infrastructure for tests to execute successfully (which is quite a bit
> > more than just 'make test')
> >
> > The need to use the most recent version of a KDE project means either:
> > a) You have to have all of those projects use the same image as well; or
> > b) You have a special job to build all of those projects that don't
> > use the same image (which is what the 'Dependency Build' jobs do); or
> > c) You incorporate the project within the base system itself
> >
> > Option (c) is not possible on Windows or FreeBSD as those platforms do
> > not support dynamic provisioning of images - meaning that all KDE
> > projects have to share the same machines and therefore must be
> > provided via our mechanisms (options a and b above).
> >
> > In the case of Plasma, baking in say Frameworks to the image would
> > mean a full image rebuild everytime you needed something new in a
> > Framework. Please note that kwindowsystem and plasma-framework are
> > both Frameworks. Suffice to say, that won't work - you would be
> > rebuilding that image multiple times a week at a minimum.
> >
> > The only way to manage the build environment for something like Plasma
> > is to do it on a Product level, taking all of it's member projects
> > into account. This is what we do currently - while the actual Docker
> > image may happen to be shared with Frameworks and Applications, that
> > isn't required and could be seperated within our existing
> > architecture.
> >
>
> Or use option (d): a repository of artifacts which you push to/pull
> from. The problem I see with that right now is: we would need "package
> management" and our projects use mostly plain CMake which by itself
> doesn't understand this concept. So the standard tooling for our
> projects can only check for declared dependencies, but cannot try to
> install them locally. As I understand it, that is effectively what a
> good chunk of the CI tooling sort of replicates anyway: a custom built
> package management solution for KDE CI.

We already do have a repository of artifacts :)
You can find the public view of this at
https://build-artifacts.kde.org/production/

That is how options (a) and (b) work - so you are right that we have a
mini package management solution of our own.
This is why we have the dependency metadata files.

>
> Out of interest, apart from the amount of work it might take what
> would be the main blocker for using VMs and things like Vagrant boxes
> for FreeBSD as the next best thing to containers? I'd ask the same for
> Windows, but I imagine the answer is licensing costs.
>

We already use Virtual Machines for both FreeBSD and Windows.

The main problem here is that there is no way of dynamically
provisioning full VMs without building an entire Openstack type
system, which is just too much overhead for the number of builders we
have.

> >
> > That isn't CI. That is CD - as it results in an artifact for use on
> > developers/testers/end user systems.
> >
>
> In the sense that if you were to push to a package repository it
> entails a deployment. But building binaries by itself obviously isn't,
> even if you expose these as downloadable artifacts.
>
> >
> > The infrastructure for Android builds has grown organically, and I
> > suspect we are at a point where we do need to take a step back and ask
> > ourselves if there is a better way of doing this that is more
> > maintainable. The dependencies bit in particular definitely needs
> > work.
> >
>
> Agreed. And I understand that there is always something to be said for
> working code now, rather than possibly working something maybe
> sometime in the future.
>
> >
> > CI is definitely not easy. Especially not when you are dealing with an
> > organisation the size of KDE, where projects depend on the latest
> > development version of other projects.
> >
>
> Oh I agree. I'm not definitely not claiming that any of this is easy
> or obvious or that what I throw out here will not ever turn out to
> have significant downsides/issues of its own. I do want to emphasise
> that from what I have seen the CI setup do and try to solve, I kind of
> understand why things are the way they are and I appreciate that you
> are coming from a well-considered position here.
>
> >
> > You mean like the ones at
> > https://invent.kde.org/sysadmin/ci-tooling/-/tree/master/system-images
> > ?
> >
>
> Yes except that the control over what image gets used is inverted:
> instead of a pre-defined CI setup which expose a set of images that
> *will* run CI for a given project,  there is a registry/repository of
> images from which a project can choose any N. Similarly there would be
> a repository of recipes/templates for what a sane CI setup would be so
> projects would have a good foundation by default. But the key
> difference here is that the project is in control of what image
> actually runs their CI, and so can experiment freely with this during
> development without requiring sysadmin tickets for a lot of things.

Shouldn't this kind of experimentation be done on developers systems
rather than the CI system?
Please do remember that CI resources are not infinite and at times can
be quite constrained.

My main query here though is what would be achieved by allowing this
sort of thing that the existing system does not allow for?
(Ignoring the fact we don't cover feature branches - something which
is in the long term pipeline)

>
> Possibly projects would also submit custom images for consideration to
> that registry. Now based on another reply to my original post I
> understand that this last bit is currently also possible. At least if
> I understood it correctly: it is possible to submit entire new images
> for consideration and then ask for your project to be configured
> accordingly?

This isn't something that is terribly easy to do within our existing
Jenkins setup - but is theoretically possible. It requires quite a bit
of provisioning work on the Jenkins side.

The CI scripts would also need some adapting to make this sort of
setup efficient to operate with them if you were to use them (and you
probably do want to use them for running tests).
A good portion of this adapting will done as part of moving the CI
system to Gitlab.

Note however that images based upon Fedora or anything that shares
it's lineage (including CentOS and it's derivates) is strictly
prohibited and won't be accepted for inclusion.

>
> Regards,
>
> - Johan

Cheers,
Ben