flatpak CI and stable builds - Re: KDE Gear projects with failing CI (release/23.08) (12 September 2023)

Mon Sep 25 09:39:21 BST 2023

On Mon, Sep 25, 2023 at 9:43 AM Albert Astals Cid <aacid at kde.org> wrote:

> El dijous, 21 de setembre de 2023, a les 9:54:59 (CEST), Ben Cooksley va
> escriure:
> > On Thu, Sep 21, 2023 at 8:35 AM Albert Astals Cid <aacid at kde.org> wrote:
> > > El dimecres, 20 de setembre de 2023, a les 13:17:22 (CEST), Ben
> Cooksley
> > > va
> > >
> > > escriure:
> > > > On Wed, Sep 20, 2023 at 9:42 AM Albert Astals Cid <aacid at kde.org>
> wrote:
> > > > > El dimarts, 19 de setembre de 2023, a les 22:18:40 (CEST),
> > > > >
> > > > > christoph at cullmann.io va escriure:
> > > > > > On 2023-09-19 21:35, Albert Astals Cid wrote:
> > > > > > > Please work on fixing them, otherwise i will remove the
> failing CI
> > > > > > > jobs on their 4th failing week, it is very important that CI is
> > > > > > > passing
> > > > > > > for
> > > > > > > multiple reasons.
> > > > > > >
> > > > > > > Bad news: We have 2 new repositories failing :/
> > > > > > >
> > > > > > > = FLATPAK FAILING =
> > > > > > >
> > > > > > > kate:
> > > > > > >  * https://invent.kde.org/utilities/kate/-/pipelines/484147
> > > > > > >
> > > > > > >   * This highlights a design problem, it's building markdown
> part
> > >
> > > from
> > >
> > > > > > > master
> > > > > > > instead of from stable branch. We can manually change the
> branch,
> > >
> > > but
> > >
> > > > > > > i
> > > > > > > would
> > > > > > > prefer a solution that doesn't mean changing lots and lots of
> > >
> > > flatpak
> > >
> > > > > > > manifests when we branch.
> > > > > >
> > > > > > Hmm, yes that sounds not nice.
> > > > > >
> > > > > > But not sure how that would work without that, seems
> > >
> > >
> https://invent.kde.org/utilities/kate/-/blob/master/.flatpak-manifest.json
> > >
> > > > > ?r>
> > > > >
> > > > > > ef_type=heads
> > > > > >
> > > > > > hard codes what to fetch.
> > > > > >
> > > > > > Given one hard codes there the
> > > > > >
> > > > > >   "runtime-version": "5.15-22.08",
> > > > >
> > > > > That one is "fine", the 22.08 here it's related to the "flatpak
> kde/
> > > > > freedesktop sdk" not to Gear stuff.
> > > > >
> > > > > Yes, we will massively have to update them on master when we
> decide to
> > > > > depend
> > > > > on a new one, but it won't cause problems on the stable branches
> like
> > >
> > > the
> > >
> > > > > oner
> > > > > we're experiencing here.
> > > > >
> > > > > The problem here is
> > > > >
> > > > > {
> > > > >
> > > > >   "name": "markdownpart",
> > > > >   "buildsystem": "cmake-ninja",
> > > > >   "sources": [
> > > > >
> > > > >     {
> > > > >
> > > > >       "type": "git",
> > > > >       "url": "https://invent.kde.org/utilities/markdownpart.git"
> > > > >
> > > > >     }
> > > > >
> > > > >   ]
> > > > >
> > > > > }
> > > > >
> > > > > This unconditionally compiles the master branch of markdownpart
> repo
> > > > >
> > > > > As far as i can see there's three solutions:
> > > > >
> > > > > A) If this is just "to make sure it builds CI", we don't need
> > >
> > > markdownpart
> > >
> > > > > nor
> > > > > konsole on the flatpak since they are just runtime dependencies.
> This
> > >
> > > is a
> > >
> > > > > sub-optimal solution i'd say since it makes it so that we can't
> offer
> > >
> > > the
> > >
> > > > > package for testing in the future and makes the diff with the
> flathub
> > > > > manifest
> > > > > bigger than it needs to be
> > > >
> > > > The overall intention is for the bundles produced by the Flatpak
> jobs to
> > >
> > > be
> > >
> > > > useful for people to locally test a project build.
> > > > In the not too distant future we'll have them available from a
> Flatpak
> > > > repository for actual mainline/release branches as well.
> > > >
> > > > So the answer certainly isn't (a).
> > > >
> > > > > B) Depend on released versions, i.e. a tarball in "sources"
> instead of
> > >
> > > a
> > >
> > > > > git
> > > > > repo. This is probably suboptimal too in the sense that will
> require
> > > > > constant
> > > > > updating on master and if we offer the resulting flatpak as
> "nightly"
> > >
> > > in
> > >
> > > > > the
> > > > > future for testing it's not "nightly" as it could be.
> > > >
> > > > Guess it depends on the nature of the dependency, but in the case of
> > > > software released together you probably want to build against what
> you
> > >
> > > will
> > >
> > > > be shipping against yes.
> > > >
> > > > > C) Add a marker in the .json like branch: "kde-same-branch" and
> then
> > >
> > > have
> > >
> > > > > the
> > > > > code in
> > >
> > >
> https://invent.kde.org/sysadmin/ci-utilities/raw/master/gitlab-templates/f
> > >
> > > > > latpak.yml replace that "kde-same-branch" either by "master" or by
> > > > > the appropriate stable branch before actually compiling the
> flatpak. I
> > > > > think
> > > > > this would be the optimal solution but needs work.
> > > >
> > > > This is my preferred solution as well. it wouldn't be too difficult
> > > > given
> > > > we have a Python script acting as a middle-man already.
> > > > In the past we did some rewriting of the .flatpak-manifest.yml
> already.
> > > >
> > > > Depending on our requirements it may be worth tying into the same
> > > > branch-rules.yml logic that the rest of the CI system uses but this
> is
> > > > probably best answered by someone who knows the various Flatpak
> > > > manifests
> > > > we have better.
> > > >
> > > > In #flatpak:kde.org it was mentioned that this would mean that
> people
> > > > wouldn't be able to build it as easily themselves, but if we make it
> > > > well
> > > > documented (comments in the .flatpak-manifest.yml, etc) then I think
> > > > this
> > > > shouldn't be much of an issue.
> > >
> > > I had not thought of that and it's indeed not great.
> > >
> > > We could rename all the flatpak manifest that need this feature to .
> > > json.in to
> > > make it clear "it's not their final form" and that they need to be pre-
> > > processed.
> > >
> > >
> > > This does not help a lot to the "non CI user" if they want to actually
> use
> > > the
> > > manifest since they still need to pre-process it somehow :/
> >
> > At least it would make it explicit that something is required before the
> > files can be utilised elsewhere.
> >
> > Ultimately in the long term we are potentially looking at having release
> > builds we make be submitted into Flathub so this does need a solution in
> > some form or another.
>
> I don't see how that would work, for flathub we use released tarballs as
> sources for everything, not git branches, i guess we could switch to git
> tags?

Correct, we would need to use Git tags for the Flathub uploads.
The same applies to our Android / Windows builds as well.

>
>
> >
> > Realistically I think we only have two choices here:
> > a) Have our release tooling include logic that bumps/updates the
> > .flatpak-manifest.yml files to have the right branch in them; or
> > b) Have a intermediary script like flatpak-build.py do that for us
> >
> > For now the path of least resistance is probably (b) I think?
>
> Probably
>
> Cheers,
>   Albert
>

Thanks,
Ben

>
> >
> > > Solution E)
> > >
> > > Since usually/mostly we have our projects be backwards API compatible
> it's
> > > usually/mostly not a problem to that kate stable uses markdown master
> > > (understanding the flatpaks generated by that CI are nightlies), we're
> > > only
> > > having this problem right now because of the KF6 transition.
> > >
> > > One potentially valid solution is to just disable flatpak CI in stable
> > > until
> > > this gets fixed, it's not great but it's just a few months.
> > >
> > > Cheers,
> > >
> > >   Albert
> >
> > Cheers,
> > Ben
> >
> > > > > D) Something smarter I have not thought about.
> > > > >
> > > > > Cheers,
> > > > >
> > > > >   Albert
> > > >
> > > > Cheers,
> > > > Ben
> > > >
> > > > > > I assume one will need to hard code that, too, if one creates no
> own
> > > > > > scripting.
> > > > > >
> > > > > > But I might be wrong.
> > > > > >
> > > > > > Greetings
> > > > > > Christoph
> > > > > >
> > > > > > > = FAILING UNIT TESTS =
> > > > > > >
> > > > > > > konsole:
> > > > > > >  * https://invent.kde.org/utilities/konsole/-/pipelines/484148
> > > > > > >
> > > > > > >   * freebsd_qt515 tests are failing
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.kde.org/pipermail/kde-devel/attachments/20230925/f71d336f/attachment-0001.htm>