CI talk (Was: re: Manner in which kde-gtk-config development is conducted)

[Johan Ouwerkerk]

On Sat, Mar 21, 2020 at 10:27 PM Ben Cooksley <bcooksley at kde.org> wrote:
>
> On Sun, Mar 22, 2020 at 3:27 AM Johan Ouwerkerk <jm.ouwerkerk at gmail.com> wrote:
> >
> > On Sat, Mar 21, 2020 at 1:32 AM Ben Cooksley <bcooksley at kde.org> wrote:
> > >
> > > Comments welcome. Please note that simply fixing the dependency
> > > breakage in this case is not enough to resolve this - there are
> > > underlying issues which need to be addressed here.
> > >
> > > Regards,
> > > Ben Cooksley
> > > KDE Sysadmin
> >
> > I cannot comment as to whether or not this is a pattern of behaviour
> > or just a few isolated instances. From a technical perspective I feel
> > there are two (additional) underlying issues worth addressing here:
> >
> >  1. This could be prevented for the most part by having CI run before,
> > and not after the fact. I.e. prior to merging code.
>
> This will happen once we are on Gitlab.
>
> >  2. Different projects have different CI needs, and it would help if a
> > project could safely manage their CI environment "on their own" as
> > much as possible. The current system requires a lot of daunting
> > (possibly otherwise unnecessary) complexity purely to manage the fact
> > that a builder image will be used not just for one project but for
> > perhaps the whole of KDE.
>
> The global builder image doesn't cause too many issues fortunately and
> isn't the problem here. Our current setup actually has the flexibility
> to use different images - we're just constrained in terms of setting
> this up by Jenkins as there is a bunch of tooling needed to provision
> jobs.
>
> A good proportion of the current "complexity" is due to our use of
> Jenkins. The remainder is due to needing to provide the most recent
> version of a given KDE project as a dependency, along with the
> infrastructure for tests to execute successfully (which is quite a bit
> more than just 'make test')
>
> The need to use the most recent version of a KDE project means either:
> a) You have to have all of those projects use the same image as well; or
> b) You have a special job to build all of those projects that don't
> use the same image (which is what the 'Dependency Build' jobs do); or
> c) You incorporate the project within the base system itself
>
> Option (c) is not possible on Windows or FreeBSD as those platforms do
> not support dynamic provisioning of images - meaning that all KDE
> projects have to share the same machines and therefore must be
> provided via our mechanisms (options a and b above).
>
> In the case of Plasma, baking in say Frameworks to the image would
> mean a full image rebuild everytime you needed something new in a
> Framework. Please note that kwindowsystem and plasma-framework are
> both Frameworks. Suffice to say, that won't work - you would be
> rebuilding that image multiple times a week at a minimum.
>
> The only way to manage the build environment for something like Plasma
> is to do it on a Product level, taking all of it's member projects
> into account. This is what we do currently - while the actual Docker
> image may happen to be shared with Frameworks and Applications, that
> isn't required and could be seperated within our existing
> architecture.
>

Or use option (d): a repository of artifacts which you push to/pull
from. The problem I see with that right now is: we would need "package
management" and our projects use mostly plain CMake which by itself
doesn't understand this concept. So the standard tooling for our
projects can only check for declared dependencies, but cannot try to
install them locally. As I understand it, that is effectively what a
good chunk of the CI tooling sort of replicates anyway: a custom built
package management solution for KDE CI.

Out of interest, apart from the amount of work it might take what
would be the main blocker for using VMs and things like Vagrant boxes
for FreeBSD as the next best thing to containers? I'd ask the same for
Windows, but I imagine the answer is licensing costs.

>
> That isn't CI. That is CD - as it results in an artifact for use on
> developers/testers/end user systems.
>

In the sense that if you were to push to a package repository it
entails a deployment. But building binaries by itself obviously isn't,
even if you expose these as downloadable artifacts.

>
> The infrastructure for Android builds has grown organically, and I
> suspect we are at a point where we do need to take a step back and ask
> ourselves if there is a better way of doing this that is more
> maintainable. The dependencies bit in particular definitely needs
> work.
>

Agreed. And I understand that there is always something to be said for
working code now, rather than possibly working something maybe
sometime in the future.

>
> CI is definitely not easy. Especially not when you are dealing with an
> organisation the size of KDE, where projects depend on the latest
> development version of other projects.
>

Oh I agree. I'm not definitely not claiming that any of this is easy
or obvious or that what I throw out here will not ever turn out to
have significant downsides/issues of its own. I do want to emphasise
that from what I have seen the CI setup do and try to solve, I kind of
understand why things are the way they are and I appreciate that you
are coming from a well-considered position here.

>
> You mean like the ones at
> https://invent.kde.org/sysadmin/ci-tooling/-/tree/master/system-images
> ?
>

Yes except that the control over what image gets used is inverted:
instead of a pre-defined CI setup which expose a set of images that
*will* run CI for a given project,  there is a registry/repository of
images from which a project can choose any N. Similarly there would be
a repository of recipes/templates for what a sane CI setup would be so
projects would have a good foundation by default. But the key
difference here is that the project is in control of what image
actually runs their CI, and so can experiment freely with this during
development without requiring sysadmin tickets for a lot of things.

Possibly projects would also submit custom images for consideration to
that registry. Now based on another reply to my original post I
understand that this last bit is currently also possible. At least if
I understood it correctly: it is possible to submit entire new images
for consideration and then ask for your project to be configured
accordingly?

Regards,

- Johan