<div dir="ltr"><div dir="ltr">On Fri, Aug 25, 2023 at 1:47 AM Scarlett Moore <<a href="mailto:scarlett.gately.moore@gmail.com" target="_blank">scarlett.gately.moore@gmail.com</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi all,<br></blockquote><div><br></div><div>Hi Scarlett,</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
First I apologize for starting this discussion without a clear intent.<br>
I thought it was a simple request as flatpak manifests are already in<br>
repos, why not snapcraft files.<br></blockquote><div><br></div><div>I think we're all pretty okay with Snapcraft files being in the repositories, however the expectation was that this would result in us getting CI for Snaps.</div><div>This was the gap that was not communicated well.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
SO first here are the problems I am trying to solve.<br>
<br>
a) I ( or any future snapcrafter snapping KDE applications ) would<br>
like a closer relationship with developers and their apps/ snaps.<br></blockquote><div><br></div><div>Makes sense.</div><div><br></div><div>To give a bit of background to how we ended up with .kde-ci.yml, .flatpak-manifest.yml and soon to be .craft.ini files in our repositories, one of the big issues we've faced with CI prior to Gitlab was how to version things like dependencies and other settings.</div><div><br></div><div>Storing this information outside the repository ( within the CI system itself ) means we have to carry around all the variants for the various versions in use, and ensure that we design in support to handle that.</div><div>This never worked particularly well, and with Gitlab having support for an unlimited set of branch variants, a different approach was needed - hence why they ended up going into the repository itself (which has a number of other benefits as well, like letting developers update it all in one MR when making a change that changes dependencies, etc)</div><div><br></div><div>This only works though when the necessary recipe (.flatpak-manifest.yml, .kde-ci.yml) refers to the sources in the folder that it sits in though - and not to some other copy such as a tarball.</div><div><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
b) Automated builds and publishing via launchpad.<br>
1) Launchpad does not support our current all in one repo in folders setup.<br>
2) Our current method for builds sends off the snapcraft files to<br>
launchpad via a public remote build in which launchpad creates a<br>
temporary snap recipe, does the build, and sends back the snaps and<br>
our CI then uploads them to the store. This is clunky at best for many<br>
reasons. Those temporary snap recipes are last on the priority list<br>
for launchpad builders and this results in many failed builds due to<br>
timeouts. In this thread there are complaints about traffic ( this<br>
setup creates loads of traffic sending the files to launchpad and the<br>
receiving sometimes large snaps in return, and then again sending them<br>
out to the store! Having launchpad do all this would reduce traffic<br>
immensely.<br></blockquote><div><br></div><div>This is quite a different topic though, and is where I think things have gotten confused.</div><div><br></div><div>For Flatpaks, Android APKs, Windows appx we'll be making use of a special process to allow for Continuous Delivery processes to be triggered from Gitlab.</div><div>It's yet to be fully determined how they'll be triggered.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
3) Support for more architectures. Having launchpad handle the<br>
builds/upload would allow for many architectures without any effort on<br>
our end. Many users ( especially of our SDK ) have requested this.<br>
c) Eventually CI UI testing on the snaps.<br>
<br>
I do not have a clear plan on how I am to achieve the technical path<br>
as of yet as I am in the R&D phase and using blinken as my test<br>
subject. So far my testing has positive results on launchpad mirroring<br>
the github repo and automatic builds on changes which looks like this:<br>
branch -> publish store channel<br>
master -> edge<br>
release/xx.xx -> candidate<br>
They would stay in candidate until the actual release happens, then as<br>
I do now - I test the stable release snap and release to the stable<br>
channel. Obviously this process is time consuming and I would like to<br>
automate it via CI UI testing.<br>
<br>
I am getting many conflicts in this discussion as to building per<br>
commit or release only. With launchpad doing all the builds /<br>
publishing the KDE infrastructure will have no impact as I have<br>
launchpad polling the github mirror. So the answer here is I will be<br>
doing both.<br>
<br>
I did not know flatpak manifests are duplicated ( on build-bot and in<br>
repos? ) Is there something in place that syncs them?<br>
The snap store does have a similiar feature with github builds but<br>
this would require github repos for the snapcraft files and I am not<br>
sure duplication is the best path forward? Especially since the<br>
launchpad method above works.<br></blockquote><div><br></div><div>For Flatpak, we have our ones that are used for CI and nightly builds.</div><div>There are also release versions though that live in the Flathub repositories and publish to Flathub. </div><div><br></div><div>I'm not aware of anything to "sync" these.<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
I am not sure I understand the need to build a snap on our CI that no<br>
one can touch? Seems using the launchpad API to check the status would<br>
be easier, but if the docker build is a requirement to move forward,<br>
it is supported <a href="https://forum.snapcraft.io/t/build-on-docker/4158" rel="noreferrer" target="_blank">https://forum.snapcraft.io/t/build-on-docker/4158</a></blockquote><div><br></div><div>You would only need to build a Snap within Docker if you were going to do CI.</div><div><br></div><div>It seems however that your objective is not CI - it is CD - so that entire point can be ignored.<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br>
<br>
<br>
I hope this clears up any questions or concerns.<br>
Scarlett<br></blockquote><div><br></div><div>Cheers,</div><div>Ben</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
<br>
On Thu, Aug 24, 2023 at 3:33 AM Ben Cooksley <<a href="mailto:bcooksley@kde.org" target="_blank">bcooksley@kde.org</a>> wrote:<br>
><br>
> On Thu, Aug 24, 2023 at 5:00 AM Albert Astals Cid <<a href="mailto:aacid@kde.org" target="_blank">aacid@kde.org</a>> wrote:<br>
>><br>
>> El dimecres, 23 d’agost de 2023, a les 18:29:10 (CEST), Scarlett Moore va escriure:<br>
>> > After some testing and various complicated options, the easiest I have<br>
>> > found is importing our github mirror in launchpad ( this avoids the<br>
>> > polling on kde servers ) and we can create our necessary snap recipes<br>
>> > and KDE infrastructure is unaffected save the already in place github<br>
>> > mirror. Launchpad does everything else ( building and uploading to<br>
>> > store ) I see no reason not to move forward as KDE infrastructure will<br>
>> > have much less involvement in the snap lifecycle aside from the<br>
>> > snapcraft files being in the application repository. As mentioned<br>
>> > before this will allow developers closer involvement with snap<br>
>> > developers.<br>
>><br>
>> Honestly I'm still a bit lost on what is you are trying to do regarding Snap and the KDE repos :D<br>
><br>
><br>
> I have to admit I am similarly confused.<br>
><br>
> I thought this was originally about CI, which is why I proposed tying this in through .gitlab-ci.yml.<br>
> Using that mechanism if this is really about final release builds doesn't make much sense.<br>
><br>
> Having a look at what was committed to Blinken's repository though, i'm not sure there is much that would stop this being actual CI (assuming someone got it working within Docker)?<br>
><br>
>><br>
>> Are we going for CI or CD?<br>
>><br>
>> CI -> snap is compiled on each commit and MR to make sure we don't break the snap build<br>
>><br>
>> CD -> We have an almost-automated way of doing snap "release" builds.<br>
>><br>
>><br>
>> To give some examples:<br>
>><br>
>> This is arianna's KDE flatpak CI<br>
>> <a href="https://invent.kde.org/graphics/arianna/-/jobs/1142328" rel="noreferrer" target="_blank">https://invent.kde.org/graphics/arianna/-/jobs/1142328</a><br>
>> It builds arianna using flatpak with each commit but the artifacts are "unused"<br>
>><br>
>><br>
>> This is Okular's KDE Windows CI<br>
>> <a href="https://invent.kde.org/graphics/okular/-/jobs/1142178" rel="noreferrer" target="_blank">https://invent.kde.org/graphics/okular/-/jobs/1142178</a><br>
>> It builds each commit to make sure the Windows build doesn't get broken<br>
>><br>
>><br>
>> This is Okular's KDE Windows CD<br>
>> <a href="https://binary-factory.kde.org/job/Okular_Release_win64/" rel="noreferrer" target="_blank">https://binary-factory.kde.org/job/Okular_Release_win64/</a><br>
>> It builds release Okular nightly in a way that is a release build and can be easily uploaded to the Windows Store<br>
>><br>
>><br>
>> This is Okular flathub CD<br>
>> <a href="https://github.com/flathub/org.kde.okular" rel="noreferrer" target="_blank">https://github.com/flathub/org.kde.okular</a><br>
>> <a href="https://buildbot.flathub.org/#/apps/org.kde.okular" rel="noreferrer" target="_blank">https://buildbot.flathub.org/#/apps/org.kde.okular</a><br>
>> It is a convenient way to generate new builds each time a stable release happens<br>
>><br>
>><br>
>> I see you committed this<br>
>> <a href="https://invent.kde.org/education/blinken/-/blob/master/.snapcraft.yaml" rel="noreferrer" target="_blank">https://invent.kde.org/education/blinken/-/blob/master/.snapcraft.yaml</a><br>
>><br>
>><br>
>> What is that going to give us?<br>
>><br>
>><br>
>> Cheers,<br>
>> Albert<br>
><br>
><br>
> Cheers,<br>
> Ben<br>
><br>
>><br>
>><br>
>><br>
>> > Thanks,<br>
>> > Scarlett<br>
>> ><br>
>> > On Sun, Aug 20, 2023 at 7:20 PM Justin Zobel <<a href="mailto:justin.zobel@gmail.com" target="_blank">justin.zobel@gmail.com</a>> wrote:<br>
>> > > I think there is a lot of misunderstanding here.<br>
>> > ><br>
>> > > In GitLab CI only test builds are done and artifacts are kept so people<br>
>> > > can test an AppImage or Flatpak without having to compile locally.<br>
>> > ><br>
>> > > Stable releases are done by the Release Team via scripts to tarballs.<br>
>> > > Flatpaks are done via Flathub and are done (usually) automatically via<br>
>> > > the Flatpak External Data Checker.<br>
>> > ><br>
>> > > Hopefully this clarifies the situation.<br>
>> > ><br>
>> > > On 21/8/23 08:48, Scarlett Moore wrote:<br>
>> > > > On Sun, Aug 20, 2023, 4:14 PM Julius Künzel<br>
>> > > ><br>
>> > > > <<a href="mailto:jk.kdedev@smartlab.uber.space" target="_blank">jk.kdedev@smartlab.uber.space</a>> wrote:<br>
>> > > > To me it seems this discussion is quite abstract and there are<br>
>> > > > several misunderstandings because not everybody knows every detail<br>
>> > > > of snap packaging and/or the KDE infrastructure (me neither).<br>
>> > > ><br>
>> > > > I understand that from an organizational perspective most people<br>
>> > > > like the idea of having the files in the repos, but have technical<br>
>> > > > doubts. Hence, I wonder whether it would be a good idea to take<br>
>> > > > one KDE app to try and showcase this suggestion?<br>
>> > > ><br>
>> > > > Cheers,<br>
>> > > > Julius<br>
>> > > ><br>
>> > > > Sounds like a great idea to me.<br>
>> > > > Scarlett<br>
>> > > ><br>
>> > > > 20.08.2023 17:55:24 Laura David Hurka <<a href="mailto:david.hurka@mailbox.org" target="_blank">david.hurka@mailbox.org</a>>:<br>
>> > > > > On Sunday, August 20, 2023 12:47:10 PM CEST Ben Cooksley wrote:<br>
>> > > > >> On Sun, Aug 20, 2023 at 12:43 PM Scarlett Moore <<br>
>> > > > >><br>
>> > > > >> <a href="mailto:scarlett.gately.moore@gmail.com" target="_blank">scarlett.gately.moore@gmail.com</a>> wrote:<br>
>> > > > >>> Only on release! We will not be building from master! We don't<br>
>> > > ><br>
>> > > > want<br>
>> > > ><br>
>> > > > >>> unstable snaps.<br>
>> > > > >>> Thanks,<br>
>> > > > >>> Scarlett<br>
>> > > > >><br>
>> > > > >> In that particular case the jobs should be manually triggered<br>
>> > > > >> only.<br>
>> > > > >><br>
>> > > > >> Gitlab CI is really made for building artifacts for a given<br>
>> > > ><br>
>> > > > commit rather<br>
>> > > ><br>
>> > > > >> than for a specified version though, so this is definitely<br>
>> > > ><br>
>> > > > going to be a<br>
>> > > ><br>
>> > > > >> case of things not fitting quite right.<br>
>> > > > >><br>
>> > > > >> Cheers,<br>
>> > > > >> Ben<br>
>> > > > >> [...]<br>
>> > > > ><br>
>> > > > > This confuses me too.<br>
>> > > > > It seems Scarlett wants to use a “deploy” stage [1] and a job<br>
>> > > ><br>
>> > > > rule [2]<br>
>> > > ><br>
>> > > > > to run snap build&release jobs automatically when the release is<br>
>> > > ><br>
>> > > > done.<br>
>> > > ><br>
>> > > > > If you mean that Gitlab CI should not be used to automate<br>
>> > > ><br>
>> > > > release jobs,<br>
>> > > ><br>
>> > > > > you should elaborate more how binary-factory is meant to be<br>
>> > > ><br>
>> > > > replaced.<br>
>> > > ><br>
>> > > > > Otherwise, do you just note that Gitlab CI is suboptimal,<br>
>> > > > > or do you recommend to use something else?<br>
>> > > > > Like: “Release build: automatic is fine. Release publish: please<br>
>> > > ><br>
>> > > > only manual”?<br>
>> > > ><br>
>> > > > > Cheers, David<br>
>> > > > ><br>
>> > > > ><br>
>> > > > > [1] <a href="https://docs.gitlab.com/ee/ci/yaml/#stages" rel="noreferrer" target="_blank">https://docs.gitlab.com/ee/ci/yaml/#stages</a><br>
>> > > > > [2] like this:<br>
>> > > > ><br>
>> > > > > snap-release-job:<br>
>> > > > > rules:<br>
>> > > > > -if: $CI_COMMIT_TAG =~ /^v[0-9][0-9]\.[0-9][0-9]\.[0-9][0-9]$/<br>
>> > > > ><br>
>> > > > > [...]<br>
>> > > > ><br>
>> > > > > see also:<br>
>> > > > <a href="https://docs.gitlab.com/ee/ci/jobs/job_control.html#use-predefined-c" rel="noreferrer" target="_blank">https://docs.gitlab.com/ee/ci/jobs/job_control.html#use-predefined-c</a><br>
>> > > > icd-variables-to-run-jobs-only-in-specific-pipeline-types<br>
>><br>
>><br>
>><br>
>><br>
</blockquote></div></div>