Debugging KDE CI runners

[Ben Cooksley]

On Wed, Oct 12, 2022 at 2:51 AM Steven Robbins <steve at sumost.ca> wrote:

> On Tuesday, October 11, 2022 3:20:06 A.M. CDT Ben Cooksley wrote:
> > On Tue, Oct 11, 2022 at 6:49 PM Steven Robbins <steve at sumost.ca> wrote:
>
> > > I am trying to understand how libraries are selected for installation
> on
> > > the
> > > KDE CI/CD machines.
>
> > There are a couple of things here to pull apart.
> >
> > First, the Gitlab CI jobs you mention above themselves don't use Craft at
> > all at build time, so the above configurations you are referring to have
> no
> > impact whatsoever.
>
> Ah, that's one crucial detail, thanks!
>
> > Those jobs rely on KDE specific Docker containers on Linux and Windows,
> and
> > on FreeBSD is a fixed statically provisioned machine.
> > For Linux and FreeBSD, we use only distribution provided packages for
> those
> > systems - while on Windows we use Craft to deploy a static and
> > centrally managed list of packages on the system.
>
> OK.  Can you help me understand the full scope of Craft?  Is it solely for
> the
> windows CI configuration?  I've heard that there are also distribution
> builders
> (for flatpack and the like) -- is Craft involved in any of that?
>

There are two parts of our systems:
a) Continuous Integration, which runs unit tests, code quality checks and
monitors code to ensure it compiles
b) Continuous Delivery, which produces binaries for end users to use.

Prior to the move to Gitlab, (a) was handled by build.kde.org, while (b)
was handled by binary-factory.kde.org.
While the move to GItlab for (a) is complete, it has yet to be completed
for (b) but will be in the next few months.

Craft is for the most part only involved in (b).

The only places Craft is used in (a) is to produce Docker images for
Windows and Android which the system uses to carry out its CI runs.
These image builds are always using a static configuration which is
centrally maintained and cannot be varied on a per-project basis.

With regards to the production of Binaries, Craft is used for the following
types of builds:
(a) Windows installers, portable archives and store upload bundles
(b) Android APKs
(c) Linux Appimages
(d) macOS *.app / DMG bundles

Linux Flatpak bundles are built using their own tooling, which is what the
existing Gitlab Flatpak job we already have in place does.


>
> > You are therefore subject to whatever versions of ffmpeg are provided by
> > OpenSUSE (our Linux Docker image provider) and FreeBSD for those
> platforms,
> > and on Windows whatever Craft has specified as the default version of
> > ffmpeg.
>
> OK -- this explains windows I suppose, as the default ffmpeg is indeed set
> to
> 5.0.1 [1].  Is this the default setting you refer to?
>
> [1]
> https://invent.kde.org/packaging/craft-blueprints-kde/-/blob/master/libs/
> ffmpeg/ffmpeg.py#L25
> <https://invent.kde.org/packaging/craft-blueprints-kde/-/blob/master/libs/ffmpeg/ffmpeg.py#L25>


Yes, that is the setting.


>
>
> For OpenSUSE and FreeBSD: is there a central location I can check to
> understand what version of suse/freebsd is being presently used?  How
> often
> are OS updates done?  Is there an annoucement list to track this?
>

The SUSE images are updated periodically on an as-needed basis to bring in
updates or additional packages we need.
There is no set schedule for this. (same applies for the other Docker
images we have)

You can view the build logs for all our Docker images (covering LInux,
Android and Windows) in the CI section of
https://invent.kde.org/sysadmin/ci-images/
Our Dockerfile definitions are hosted in that same repository.

For FreeBSD, those VMs are maintained by the KDE FreeBSD team on Virtual
Machines provided by KDE Sysadmin.
These are maintained manually, and once again there is no schedule for this.


>
> Now comes my real question.  At present, Digikam aims to support both
> FFMPEG 4
> and 5.  Therefore it is best to test both configurations.  Is there any
> mechanism to have TWO CI runners per OS -- one with each version of ffmpeg?
>

Sorry, but it is unnecessary to have two builds per operating system when
you will get just as good coverage by having Linux build with ffmpeg 4 and
Windows build with ffmpeg 5.
We adopt the same process for testing compiler coverage, with Linux
handling GCC, FreeBSD handling Clang and Windows handling MSVC.

Additionally, Digikam is one of three projects that consume significant
amounts of CI resources, so i'd very much rather that didn't grow further.


>
>
> > With regards to your Craft Blueprint above, my understanding is that what
> > you have there is not valid syntax, as Craft doesn't look at individual
> > blueprints to resolve versions to use.
> > This is why in your Binary Factory configuration you have the following
> > specified:
> >
> > 'Digikam':
> >  buildBlueprint: "digikam"
> >  versions:
> >  - name: 'Nightly'
> >    target: 'master'
> >    packageAppx: True
> >    options:
> >    - 'libs/ffmpeg.version=4.4'
> >  platforms:
> >  - 'macos'
> >  - 'win64'
> >  - 'mingw64'
> >  - 'appimage-centos7'
> >  - 'android-x86_64'
>
> OK, but then is anything actually adhering to "libs/ffmpeg.version=4.4"?
> Clearly the windows CI runner is not.
>

This is from your Binary Factory build configuration, which is the (b) I
mentioned above and quite separate and different to (a).


>
> Thanks very much!
> -Steve
>

Regards,
Ben
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.kde.org/pipermail/kde-devel/attachments/20221012/ffda605a/attachment.htm>