Future of Android binary factory builds

[Aleix Pol]

On Sat, Dec 14, 2019 at 1:08 AM Ben Cooksley <bcooksley at kde.org> wrote:
>
> On Sat, Dec 14, 2019 at 12:29 AM Nicolas Fella <nicolas.fella at gmx.de> wrote:
> >
> > I think Ben's concern is the load on our servers that is created by rebuilding each framework for each application build.
>
> Both. Ideally we don't want the builds to take any longer than they
> really should, while at the same time maximising the scalability of
> the system to handle all our builds (so we don't end up needing a
> large farm of machines to support a comparatively small number of
> builds).
>
> We utilise this approach with Krita's dependencies on Windows, MacOS
> and Linux Appimages, where all the dependencies are built in a
> separate set of dependency jobs and then reused by the application
> build jobs.
>
> Obviously we wouldn't have a dependency job for every single Android
> one, but a single shared one for all of them (being one per
> architecture and build type - release/debug) should be more than
> sufficient to cut down on the amount of stuff being built.
>
> >
> > Nico
>
> Cheers,
> Ben
>
> >
> > On 13 December 2019 11:03:06 CET, Aleix Pol <aleixpol at kde.org> wrote:
> >>
> >> On Thu, Dec 12, 2019 at 7:35 PM Nicolas Fella <nicolas.fella at gmx.de> wrote:
> >>>
> >>>
> >>>  Hi,
> >>>
> >>>  this is a breakout of https://mail.kde.org/pipermail/release-team/2019-December/011702.html since it's not really relevant for the release team.
> >>>
> >>>  The Android binary factory currently serves two purpose. It serves as a CI
> >>>  system in the sense that it catches (compiletime) breakages and it provides
> >>>  nightly APKs for beta users.
> >>>
> >>>  Given that we already have a few applications on Google Play and it's likely
> >>>  that more will join in the future I would like to expand the scope of it to
> >>>  also provide release builds from it, i.e. from stable branches, with binaries
> >>>  built in release mode etc. This aims to replace the custom solutions some
> >>>  projects have built for themselves.
> >>>
> >>>  On Donnerstag, 12. Dezember 2019 18:54:28 CET Ben Cooksley wrote:
> >>>>
> >>>>  With the way Android builds are currently conducted (building every
> >>>>  single dependency from scratch each time) i'm opposed to doing this as
> >>>>  it would double what is already quite an expensive load on the system
> >>>>  already.
> >>>>
> >>>>  I'd like to see the Android build process optimised to let it reuse
> >>>>  the results for most dependencies (Frameworks come to mind in
> >>>>  particular) before we head in that direction.
> >>>
> >>>
> >>>  Assuming that the general approach of using Docker for the builds is fine I
> >>>  would like to propose this:
> >>>  We provide a base image that contains the most common depenencies prebuilt,
> >>>  i.e. Qt and the main Frameworks. We provide this in two variants, stable and
> >>>  unstable. Stable would provide the latest frameworks release and would thus be
> >>>  regularly rebuilt once a month after the Frameworks release. The unstable
> >>>  version would contain a master Frameworks build to keep the CI/nightly aspect
> >>>  of the builds. This would be rebuilt more often, i.e. once a day. This would
> >>>  require a full Frameworks build per day which is still an improvement over 20
> >>>  full Frameworks builds per day as we have now. With regard to the Qt version I
> >>>  think sticking to the latest release as we do now is fine for both. A solution
> >>>  that would not need a full Qt build once a day would thus be preferrable, e.g.
> >>>  by using prebuilt Qt binaries or factoring out the Qt build into another base
> >>>  image. Application builds then would pick an appropriate base image, build the
> >>>  non-framework dependencies and finally the application itself. This should
> >>>  reduce the non-necessary frameworks rebuilds by a significant amount.
> >>>
> >>>  @Ben do you agree with this proposal?
> >>
> >>
> >> Am I right in the understanding that the reason to do this is we want
> >> faster builds on binary-factory?
> >>
> >> Aleix
> >
> >
> > --
> > Sent from my Android device with K-9 Mail. Please excuse my brevity.

Hi,
I put this together:
https://invent.kde.org/sysadmin/ci-tooling/commit/b1e5400926b44f59c56366c74317bb469eb703f3

Could you please set it up to be run on build.kde.org? I haven't had
time to do a proper testing round because I'm travelling today but it
should just work.

Aleix