KDE Gear projects with failing CI (master) (30 July 2024)

[Ben Cooksley]

On Wed, Jul 31, 2024 at 7:24 PM Ingo Klöcker <kloecker at kde.org> wrote:

> On Mittwoch, 31. Juli 2024 00:05:55 CEST Albert Astals Cid wrote:
> > neochat - 2nd week
> >  * https://invent.kde.org/network/neochat/-/pipelines/744320
> >   * flatpak packaging fails
> >    * complains about signing?
>
> The verbose log in task-debug.log contains
> ```
> 2024-07-30 17:32:10,257 INFO flatpaksigner Running ['flatpak',
> 'build-import-
> bundle', '--update-appstream', '--gpg-
> sign=C78BF89D05C413A086459241700327D7B23406B3', '/tmp/tmpxyj846m1/neochat-
> nightly', 'unsigned.flatpak'] ...
> 2024-07-30 17:32:11,686 ERROR flatpaksigner The process exited with 1
> 2024-07-30 17:32:11,687 DEBUG flatpaksigner stdout:
> b'Importing app/org.kde.neochat/x86_64/master
> (27153ab08ced81594894c186cbf4ba62d90b979b708a19b78a210b3dbb818f3d)\n'
> 2024-07-30 17:32:11,687 DEBUG flatpaksigner stderr:
> b"\n(flatpak build-import-bundle:667729): GLib-CRITICAL **: 17:32:10.687:
> g_str_has_prefix: assertion 'str != NULL' failed\n\n(flatpak build-import-
> bundle:667729): OSTree-CRITICAL **: 17:32:10.687: _ostree_repo_get_remote:
> assertion 'name != NULL' failed\n\n(flatpak build-import-bundle:667729):
> GLib-
> CRITICAL **: 17:32:10.687: g_propagate_error: assertion 'src != NULL'
> failed\nerror: Executing delta part 0: opcode open-splice-and-close:
> Writing
> content object: min-free-space-percent '3%' would be exceeded, at least
> 77.8\xc2\xa0kB requested\n"
> 2024-07-30 17:32:13,130 ERROR flatpaksigner Error: Processing task
> '20240730T172831-network-neochat_2014953' failed with Error('Running
> flatpak
> failed'). Giving up.
> ```
>
> It looks like it's a disk space problem.
>
> On the signer I see
> ```
> $ df -h
> Filesystem      Size  Used Avail Use% Mounted on
> /dev/vda2        49G   35G   12G  75% /
> ```
> i.e. there are 12 GiB (~ 25 %) disk space free. While signing a Flatpak
> repo
> the repo is temporarily copied from cdn.kde.org to the signer. My guess
> is
> that the NeoChat Flatpak repo is close to 12 GiB so that less than 3 % of
> space is left which seems to be the cause of the above error.
>
> I'm not sure how best to proceed. Throw more disk space at the problem?
> Reduce
> the size of the NeoChat Flatpak repo (and of all other Flatpak repos)?
> Start a
> new NeoChat Flatpak repo from scratch?
>

I've done a couple of things now which both immediately resolve the issue
as well as move us towards a longer term solution.

First: overall disk space utilisation on the signer is due to MacOS and
Android keeping excessive logs. I have now purged those, not sure why we
had 15GB+ of dated log files between the two of them.
That has brought us up to much more reasonable free space levels:

Filesystem      Size  Used Avail Use% Mounted on
/dev/vda2        49G   24G   24G  51% /

The second step is enabling pruning of our Flatpak repositories, where only
a certain number of builds are kept - i've hardcoded this at 10 but we
could look at tweaking that or making it configurable through the project
level configuration perhaps.
See https://invent.kde.org/sysadmin/ci-notary-service/-/merge_requests/69
for that change.

That is something we had on the Binary Factory - although it had to be set
much more aggressively there due to the sheer number of builds and the
space that the full size Flatpak repository demanded.
It is probably worth checking with a Flatpak expert regarding whether the
parameters i've suggested are technically correct to ensure they won't
cause other issues (excessive repacking of Git repositories can actually
make them larger for instance)

In terms of the largest Flatpak repositories we currently have, it looks
like Neochat was the first to get >10GB although others aren't far behind:

root at tinami ~ # du -h --max-depth=1 /srv/www/cdn.kde.org/flatpak/ | grep G
12G     /srv/www/cdn.kde.org/flatpak/neochat-nightly
7.4G    /srv/www/cdn.kde.org/flatpak/gcompris-nightly
1.4G    /srv/www/cdn.kde.org/flatpak/rkward-nightly
1.3G    /srv/www/cdn.kde.org/flatpak/labplot-nightly
1.8G    /srv/www/cdn.kde.org/flatpak/umbrello-nightly
1.8G    /srv/www/cdn.kde.org/flatpak/kde-runtime-nightly
3.9G    /srv/www/cdn.kde.org/flatpak/kdenlive-nightly
1.4G    /srv/www/cdn.kde.org/flatpak/kleopatra-nightly
1.1G    /srv/www/cdn.kde.org/flatpak/kate-nightly
1.9G    /srv/www/cdn.kde.org/flatpak/itinerary-nightly
1.6G    /srv/www/cdn.kde.org/flatpak/ruqola-nightly
48G     /srv/www/cdn.kde.org/flatpak/


>
> Regards,
> Ingo


Cheers,
Ben
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.kde.org/pipermail/kde-devel/attachments/20240731/a3e01ee6/attachment-0001.htm>