Per project repository snapcraft files?
Scarlett Moore
scarlett.gately.moore at gmail.com
Thu Aug 24 14:47:13 BST 2023
Hi all,
First I apologize for starting this discussion without a clear intent.
I thought it was a simple request as flatpak manifests are already in
repos, why not snapcraft files.
SO first here are the problems I am trying to solve.
a) I ( or any future snapcrafter snapping KDE applications ) would
like a closer relationship with developers and their apps/ snaps.
b) Automated builds and publishing via launchpad.
1) Launchpad does not support our current all in one repo in folders setup.
2) Our current method for builds sends off the snapcraft files to
launchpad via a public remote build in which launchpad creates a
temporary snap recipe, does the build, and sends back the snaps and
our CI then uploads them to the store. This is clunky at best for many
reasons. Those temporary snap recipes are last on the priority list
for launchpad builders and this results in many failed builds due to
timeouts. In this thread there are complaints about traffic ( this
setup creates loads of traffic sending the files to launchpad and the
receiving sometimes large snaps in return, and then again sending them
out to the store! Having launchpad do all this would reduce traffic
immensely.
3) Support for more architectures. Having launchpad handle the
builds/upload would allow for many architectures without any effort on
our end. Many users ( especially of our SDK ) have requested this.
c) Eventually CI UI testing on the snaps.
I do not have a clear plan on how I am to achieve the technical path
as of yet as I am in the R&D phase and using blinken as my test
subject. So far my testing has positive results on launchpad mirroring
the github repo and automatic builds on changes which looks like this:
branch -> publish store channel
master -> edge
release/xx.xx -> candidate
They would stay in candidate until the actual release happens, then as
I do now - I test the stable release snap and release to the stable
channel. Obviously this process is time consuming and I would like to
automate it via CI UI testing.
I am getting many conflicts in this discussion as to building per
commit or release only. With launchpad doing all the builds /
publishing the KDE infrastructure will have no impact as I have
launchpad polling the github mirror. So the answer here is I will be
doing both.
I did not know flatpak manifests are duplicated ( on build-bot and in
repos? ) Is there something in place that syncs them?
The snap store does have a similiar feature with github builds but
this would require github repos for the snapcraft files and I am not
sure duplication is the best path forward? Especially since the
launchpad method above works.
I am not sure I understand the need to build a snap on our CI that no
one can touch? Seems using the launchpad API to check the status would
be easier, but if the docker build is a requirement to move forward,
it is supported https://forum.snapcraft.io/t/build-on-docker/4158
I hope this clears up any questions or concerns.
Scarlett
On Thu, Aug 24, 2023 at 3:33 AM Ben Cooksley <bcooksley at kde.org> wrote:
>
> On Thu, Aug 24, 2023 at 5:00 AM Albert Astals Cid <aacid at kde.org> wrote:
>>
>> El dimecres, 23 d’agost de 2023, a les 18:29:10 (CEST), Scarlett Moore va escriure:
>> > After some testing and various complicated options, the easiest I have
>> > found is importing our github mirror in launchpad ( this avoids the
>> > polling on kde servers ) and we can create our necessary snap recipes
>> > and KDE infrastructure is unaffected save the already in place github
>> > mirror. Launchpad does everything else ( building and uploading to
>> > store ) I see no reason not to move forward as KDE infrastructure will
>> > have much less involvement in the snap lifecycle aside from the
>> > snapcraft files being in the application repository. As mentioned
>> > before this will allow developers closer involvement with snap
>> > developers.
>>
>> Honestly I'm still a bit lost on what is you are trying to do regarding Snap and the KDE repos :D
>
>
> I have to admit I am similarly confused.
>
> I thought this was originally about CI, which is why I proposed tying this in through .gitlab-ci.yml.
> Using that mechanism if this is really about final release builds doesn't make much sense.
>
> Having a look at what was committed to Blinken's repository though, i'm not sure there is much that would stop this being actual CI (assuming someone got it working within Docker)?
>
>>
>> Are we going for CI or CD?
>>
>> CI -> snap is compiled on each commit and MR to make sure we don't break the snap build
>>
>> CD -> We have an almost-automated way of doing snap "release" builds.
>>
>>
>> To give some examples:
>>
>> This is arianna's KDE flatpak CI
>> https://invent.kde.org/graphics/arianna/-/jobs/1142328
>> It builds arianna using flatpak with each commit but the artifacts are "unused"
>>
>>
>> This is Okular's KDE Windows CI
>> https://invent.kde.org/graphics/okular/-/jobs/1142178
>> It builds each commit to make sure the Windows build doesn't get broken
>>
>>
>> This is Okular's KDE Windows CD
>> https://binary-factory.kde.org/job/Okular_Release_win64/
>> It builds release Okular nightly in a way that is a release build and can be easily uploaded to the Windows Store
>>
>>
>> This is Okular flathub CD
>> https://github.com/flathub/org.kde.okular
>> https://buildbot.flathub.org/#/apps/org.kde.okular
>> It is a convenient way to generate new builds each time a stable release happens
>>
>>
>> I see you committed this
>> https://invent.kde.org/education/blinken/-/blob/master/.snapcraft.yaml
>>
>>
>> What is that going to give us?
>>
>>
>> Cheers,
>> Albert
>
>
> Cheers,
> Ben
>
>>
>>
>>
>> > Thanks,
>> > Scarlett
>> >
>> > On Sun, Aug 20, 2023 at 7:20 PM Justin Zobel <justin.zobel at gmail.com> wrote:
>> > > I think there is a lot of misunderstanding here.
>> > >
>> > > In GitLab CI only test builds are done and artifacts are kept so people
>> > > can test an AppImage or Flatpak without having to compile locally.
>> > >
>> > > Stable releases are done by the Release Team via scripts to tarballs.
>> > > Flatpaks are done via Flathub and are done (usually) automatically via
>> > > the Flatpak External Data Checker.
>> > >
>> > > Hopefully this clarifies the situation.
>> > >
>> > > On 21/8/23 08:48, Scarlett Moore wrote:
>> > > > On Sun, Aug 20, 2023, 4:14 PM Julius Künzel
>> > > >
>> > > > <jk.kdedev at smartlab.uber.space> wrote:
>> > > > To me it seems this discussion is quite abstract and there are
>> > > > several misunderstandings because not everybody knows every detail
>> > > > of snap packaging and/or the KDE infrastructure (me neither).
>> > > >
>> > > > I understand that from an organizational perspective most people
>> > > > like the idea of having the files in the repos, but have technical
>> > > > doubts. Hence, I wonder whether it would be a good idea to take
>> > > > one KDE app to try and showcase this suggestion?
>> > > >
>> > > > Cheers,
>> > > > Julius
>> > > >
>> > > > Sounds like a great idea to me.
>> > > > Scarlett
>> > > >
>> > > > 20.08.2023 17:55:24 Laura David Hurka <david.hurka at mailbox.org>:
>> > > > > On Sunday, August 20, 2023 12:47:10 PM CEST Ben Cooksley wrote:
>> > > > >> On Sun, Aug 20, 2023 at 12:43 PM Scarlett Moore <
>> > > > >>
>> > > > >> scarlett.gately.moore at gmail.com> wrote:
>> > > > >>> Only on release! We will not be building from master! We don't
>> > > >
>> > > > want
>> > > >
>> > > > >>> unstable snaps.
>> > > > >>> Thanks,
>> > > > >>> Scarlett
>> > > > >>
>> > > > >> In that particular case the jobs should be manually triggered
>> > > > >> only.
>> > > > >>
>> > > > >> Gitlab CI is really made for building artifacts for a given
>> > > >
>> > > > commit rather
>> > > >
>> > > > >> than for a specified version though, so this is definitely
>> > > >
>> > > > going to be a
>> > > >
>> > > > >> case of things not fitting quite right.
>> > > > >>
>> > > > >> Cheers,
>> > > > >> Ben
>> > > > >> [...]
>> > > > >
>> > > > > This confuses me too.
>> > > > > It seems Scarlett wants to use a “deploy” stage [1] and a job
>> > > >
>> > > > rule [2]
>> > > >
>> > > > > to run snap build&release jobs automatically when the release is
>> > > >
>> > > > done.
>> > > >
>> > > > > If you mean that Gitlab CI should not be used to automate
>> > > >
>> > > > release jobs,
>> > > >
>> > > > > you should elaborate more how binary-factory is meant to be
>> > > >
>> > > > replaced.
>> > > >
>> > > > > Otherwise, do you just note that Gitlab CI is suboptimal,
>> > > > > or do you recommend to use something else?
>> > > > > Like: “Release build: automatic is fine. Release publish: please
>> > > >
>> > > > only manual”?
>> > > >
>> > > > > Cheers, David
>> > > > >
>> > > > >
>> > > > > [1] https://docs.gitlab.com/ee/ci/yaml/#stages
>> > > > > [2] like this:
>> > > > >
>> > > > > snap-release-job:
>> > > > > rules:
>> > > > > -if: $CI_COMMIT_TAG =~ /^v[0-9][0-9]\.[0-9][0-9]\.[0-9][0-9]$/
>> > > > >
>> > > > > [...]
>> > > > >
>> > > > > see also:
>> > > > https://docs.gitlab.com/ee/ci/jobs/job_control.html#use-predefined-c
>> > > > icd-variables-to-run-jobs-only-in-specific-pipeline-types
>>
>>
>>
>>
More information about the kde-devel
mailing list