Future of Android binary factory builds

[Ben Cooksley]

On Mon, Dec 23, 2019 at 3:38 PM Aleix Pol <aleixpol at kde.org> wrote:
>
> Yes, this one should work just the same as the other one, it will just
> pick the frameworks already built.

Thanks for confirming. Could we get a similar one setup for the other
architecture, after which i'll swap the CI image to derive from the
Frameworks SDK image?

Cheers,
Ben

>
> Aleix
>
> On Sat, Dec 21, 2019 at 10:43 PM Ben Cooksley <bcooksley at kde.org> wrote:
> >
> > On Mon, Dec 16, 2019 at 5:21 AM Aleix Pol <aleixpol at kde.org> wrote:
> > >
> > > On Sat, Dec 14, 2019 at 1:08 AM Ben Cooksley <bcooksley at kde.org> wrote:
> > > >
> > > > On Sat, Dec 14, 2019 at 12:29 AM Nicolas Fella <nicolas.fella at gmx.de> wrote:
> > > > >
> > > > > I think Ben's concern is the load on our servers that is created by rebuilding each framework for each application build.
> > > >
> > > > Both. Ideally we don't want the builds to take any longer than they
> > > > really should, while at the same time maximising the scalability of
> > > > the system to handle all our builds (so we don't end up needing a
> > > > large farm of machines to support a comparatively small number of
> > > > builds).
> > > >
> > > > We utilise this approach with Krita's dependencies on Windows, MacOS
> > > > and Linux Appimages, where all the dependencies are built in a
> > > > separate set of dependency jobs and then reused by the application
> > > > build jobs.
> > > >
> > > > Obviously we wouldn't have a dependency job for every single Android
> > > > one, but a single shared one for all of them (being one per
> > > > architecture and build type - release/debug) should be more than
> > > > sufficient to cut down on the amount of stuff being built.
> > > >
> > > > >
> > > > > Nico
> > > >
> > > > Cheers,
> > > > Ben
> > > >
> > > > >
> > > > > On 13 December 2019 11:03:06 CET, Aleix Pol <aleixpol at kde.org> wrote:
> > > > >>
> > > > >> On Thu, Dec 12, 2019 at 7:35 PM Nicolas Fella <nicolas.fella at gmx.de> wrote:
> > > > >>>
> > > > >>>
> > > > >>>  Hi,
> > > > >>>
> > > > >>>  this is a breakout of https://mail.kde.org/pipermail/release-team/2019-December/011702.html since it's not really relevant for the release team.
> > > > >>>
> > > > >>>  The Android binary factory currently serves two purpose. It serves as a CI
> > > > >>>  system in the sense that it catches (compiletime) breakages and it provides
> > > > >>>  nightly APKs for beta users.
> > > > >>>
> > > > >>>  Given that we already have a few applications on Google Play and it's likely
> > > > >>>  that more will join in the future I would like to expand the scope of it to
> > > > >>>  also provide release builds from it, i.e. from stable branches, with binaries
> > > > >>>  built in release mode etc. This aims to replace the custom solutions some
> > > > >>>  projects have built for themselves.
> > > > >>>
> > > > >>>  On Donnerstag, 12. Dezember 2019 18:54:28 CET Ben Cooksley wrote:
> > > > >>>>
> > > > >>>>  With the way Android builds are currently conducted (building every
> > > > >>>>  single dependency from scratch each time) i'm opposed to doing this as
> > > > >>>>  it would double what is already quite an expensive load on the system
> > > > >>>>  already.
> > > > >>>>
> > > > >>>>  I'd like to see the Android build process optimised to let it reuse
> > > > >>>>  the results for most dependencies (Frameworks come to mind in
> > > > >>>>  particular) before we head in that direction.
> > > > >>>
> > > > >>>
> > > > >>>  Assuming that the general approach of using Docker for the builds is fine I
> > > > >>>  would like to propose this:
> > > > >>>  We provide a base image that contains the most common depenencies prebuilt,
> > > > >>>  i.e. Qt and the main Frameworks. We provide this in two variants, stable and
> > > > >>>  unstable. Stable would provide the latest frameworks release and would thus be
> > > > >>>  regularly rebuilt once a month after the Frameworks release. The unstable
> > > > >>>  version would contain a master Frameworks build to keep the CI/nightly aspect
> > > > >>>  of the builds. This would be rebuilt more often, i.e. once a day. This would
> > > > >>>  require a full Frameworks build per day which is still an improvement over 20
> > > > >>>  full Frameworks builds per day as we have now. With regard to the Qt version I
> > > > >>>  think sticking to the latest release as we do now is fine for both. A solution
> > > > >>>  that would not need a full Qt build once a day would thus be preferrable, e.g.
> > > > >>>  by using prebuilt Qt binaries or factoring out the Qt build into another base
> > > > >>>  image. Application builds then would pick an appropriate base image, build the
> > > > >>>  non-framework dependencies and finally the application itself. This should
> > > > >>>  reduce the non-necessary frameworks rebuilds by a significant amount.
> > > > >>>
> > > > >>>  @Ben do you agree with this proposal?
> > > > >>
> > > > >>
> > > > >> Am I right in the understanding that the reason to do this is we want
> > > > >> faster builds on binary-factory?
> > > > >>
> > > > >> Aleix
> > > > >
> > > > >
> > > > > --
> > > > > Sent from my Android device with K-9 Mail. Please excuse my brevity.
> > >
> > > Hi,
> >
> > Hi Aleix,
> >
> > > I put this together:
> > > https://invent.kde.org/sysadmin/ci-tooling/commit/b1e5400926b44f59c56366c74317bb469eb703f3
> > >
> > > Could you please set it up to be run on build.kde.org? I haven't had
> > > time to do a proper testing round because I'm travelling today but it
> > > should just work.
> >
> > I've now setup https://build.kde.org/job/Administration/job/Docker%20Generate%20AndroidQt5.13%20AArch64%20KF5-SDK/
> >
> > I assume we would change the CI images to pull from this image rather
> > than the current main SDK image if this works out okay?
> > (Alternatively, we could fold the KF5 pre-compiled binaries into the
> > CI image if we don't think others would want to use it)
> >
> > >
> > > Aleix
> >
> > Cheers,
> > Ben