Post-MegaRelease projects

[Ben Cooksley]

On Sat, Feb 24, 2024 at 9:21 PM Volker Krause <vkrause at kde.org> wrote:

> On Saturday, 24 February 2024 04:31:49 CET Ben Cooksley wrote:
> > On Fri, Feb 23, 2024 at 11:12 PM Sune Vuorela <nospam at vuorela.dk> wrote:
> > > On 2024-02-22, Nate Graham <nate at kde.org> wrote:
> > > > I've started pondering post-megarelease projects. We've spent so
> long on
> > > > porting and bugfixing that I think it might be useful to shift gears
> to
> > > > feature work, and I'd like to brainstorm potential large-scale
> projects
> > > > and gauge the level of interest in putting resources into them soon.
> > >
> > > A bit more from the devops end that I'd love to see people tackle:
> > >  - Ensure frameworks and app unit tests interacting with windows can
> run
> > >
> > >    on Windows.
> > >    More details: The following fails on our windows CI
> > >
> https://invent.kde.org/sune/windows-test-thingie/-/blob/master/main.cpp
> > >    I find it weird that we are spending resources on putting things in
> > >    the windows store and making apps available on windows, but we can't
> > >    actually have passing tests in our CI.
> >
> > This unfortunately will not be easy to solve.
>
> And that's fine, we are in dreaming territory here anyway :)
>
> > One of the key things that we've learned out of doing CI, as has been
> > showcased by FreeBSD in particular, is that the builders need to be
> > ephemeral - that is only around for the build in question that is being
> run.
> > We're currently accomplishing this by using containers - via Podman (for
> > Linux/Android/FreeBSD) and Docker (for Windows).
> >
> > Containers also offer us the advantage of allowing people to easily
> > reproduce the CI environment on their local system without too much
> trouble.
> >
> > For Windows however, Microsoft has limited the container stack to not
> allow
> > anything GUI related to work. The underlying libraries may be there, but
> > the equivalent display server components are not operational.
> >
> > To complicate things further, on Windows certain permissions are
> restricted
> > to the interactive console and are not possible to do as either a
> scheduled
> > task or as a system service.
> > Usage of existing Windows automation frameworks such as Powershell
> Remoting
> > or SSH will therefore not work if we want things to perfectly replicate a
> > end user environment - because those will run the command(s) as part of a
> > non-interactive session (even if the user we connect as is the same one
> > logged in on the desktop console).
> >
> > Resolving this will therefore not only require running full sized Windows
> > VMs (on an ephemeral basis to avoid the recently resolved issues that
> used
> > to plague FreeBSD), but would also need the VM to automatically login and
> > spawn an agent as part of the interactive desktop session that we would
> > connect to in order to run the CI tests.
> >
> > The spawning of a VM would require refactoring the setup of our poor CI
> > workers (again - after they were only just recently completely rebuilt to
> > allow the transition to Podman to fix flatpak-builder) to make use of
> > something like:
> > -
> >
> https://forum.gitlab.com/t/custom-executor-into-windows-vm-using-linux-kvm-l
> > ibvirt/72713/5 -
> > https://docs.gitlab.com/runner/executors/custom_examples/libvirt.html
> >
> > We would still have to write the agent that receives our commands
> > (something like
> > https://gist.github.com/cschwede/3e2c025408ab4af531651098331cce45 maybe)
> >
> > We would still have to solve the issue of how to share disk images among
> > our nodes so they were built (ideally would be able to use Gitlab's
> > Container Registry for this, which is something Tart can do - Tart being
> a
> > virtualisation management utility for ARM based Macs), as well as
> > automating the installation of the various OSes we need to run this way.
> > See
> >
> https://learn.microsoft.com/en-us/windows-hardware/manufacture/desktop/autom
> > ate-windows-setup?view=windows-11 for some notes on that for Windows.
> >
> > The big downside to all of that of course is that the worker nodes would
> > take longer to startup, and it would make sharing build artifacts much
> more
> > difficult and/or less efficient.
> >
> > >  - Find a way to run unit tests on android CI.
> >
> > Likewise, this is very non-trivial - from my understanding our tooling
> > currently for building Android software is centered around it all being
> > cross compiled rather than being built on the native architecture it will
> > be run on.
> > Naturally tests cannot be run (unless you emulate, which I guess we could
> > consider...) if the CPU architectures are not compatible.
> >
> > Even if you emulate though, I imagine we would need to provide a full
> > Android system setup (including all of it's relevant bits of system
> > infrastructure, such as it's display server DisplayFlinger) in order for
> > tests to truly be of use.
> > The path of least resistance is probably by making use of Google's
> existing
> > Android emulator - although i've no idea how you would tie that into CI.
>
> Right, the Android emulator is probably the first thing to look at for
> this.
> That alone isn't enough though, we can't just copy over a unit test
> executable
> and run it there, this needs to be wrapped in an APK (and which
> potentially
> needs to be given permissions, as we don't want system UI dialogs popping
> up,
> etc).
>
> The Qt CI seems to be doing something like this meanwhile, might be worth
> investigating how this is done there.
>
> > We would need to have our build chains ready to build on a native system
> > before we could think about this I think. Building Android x86/x64
> wouldn't
> > cover things off properly due as it won't reflect the actual CPUs being
> > used on end-user devices (and ARM processors can expose issues that don't
> > happen on x86/x64 based systems)
>
> For Android I'd rate that less relevant actually, we'd get by far the most
> things covered with x86_64 as well. I'm only aware of one ARM-specific
> runtime
> issue recently, and that was ARM32 only even. The vast majority of
> problems
> are either related to APK packaging or interaction with (changing) Android
> platform infrastructure, all of which is platform-independent. And the
> majority of ARM-specific issues we actually encounter during building.
>
> ARM64 native builds nevertheless belong on this wishlist, but IMHO more
> importantly for ARM Flatpaks for the mobile apps.
>

I already have a ticket for that - https://phabricator.kde.org/T17120

Hardware will likely be provided via a CAX31 / CAX41 (
https://www.hetzner.com/cloud/) as the number of builds isn't likely to be
huge and therefore cannot justify a RX170 (
https://www.hetzner.com/dedicated-rootserver/matrix-rx/)
(which would make more sense if all our Android builds were ARM native)

At this stage i'd like to get us comfortably through into the post-Binary
Factory world before anything further takes place though (some teething
issues have popped up that weren't immediately evident prior to it's
decommissioning that will need resolving).

>From my side, the next big thing to start breaking down and building via CI
would be docs.kde.org.


>
> > >  - Make autotests guarding on all our CI's.
> > >
> > >  - Clazy and clang-tidy and cppcheck on all our repositories in CI
> > >
> > >  /Sune
> >
> > Hopefully the above is food for thought as to why we don't have the
> above -
> > don't mean to say it can't be done, but it certainly is not trivial to
> > accomplish...
>
> Sure, and likewise the wishlist isn't meant as criticism of what we have,
> on
> the contrary, IMHO our CI/CD system has made huge steps forward with the
> migration to Gitlab and to container runners as well as the new signing
> service, but now we are hooked for more ;-)
>
> Yes, those are ambitious ideas, but they would significantly help
> application
> development I think, and assuming Nate's original question was also about
> where the e.V. might want to invest resources into going forward, our
> CI/CD
> infrastructure would IMHO be a good area to (continue to) invest in.
>
> Regards,
> Volker
>

Cheers,
Ben
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.kde.org/pipermail/kde-community/attachments/20240224/da4c5a8b/attachment.htm>