Future of Android binary factory builds

[Ben Cooksley]

On Tue, Dec 24, 2019 at 3:59 AM Aleix Pol <aleixpol at kde.org> wrote:
>
> Done, should work as is as soon as the arm version is available.

Thanks Aleix, once the image builds have succeeded i'll update the
jobs to chain correctly and rebuild everything.
It looks like the Aarch64 one succeeded once and then failed on it's
next run, so the images may need tweaking.

Cheers,
Ben

>
> Aleix
>
> On Mon, Dec 23, 2019 at 9:28 AM Ben Cooksley <bcooksley at kde.org> wrote:
> >
> > On Mon, Dec 23, 2019 at 3:38 PM Aleix Pol <aleixpol at kde.org> wrote:
> > >
> > > Yes, this one should work just the same as the other one, it will just
> > > pick the frameworks already built.
> >
> > Thanks for confirming. Could we get a similar one setup for the other
> > architecture, after which i'll swap the CI image to derive from the
> > Frameworks SDK image?
> >
> > Cheers,
> > Ben
> >
> > >
> > > Aleix
> > >
> > > On Sat, Dec 21, 2019 at 10:43 PM Ben Cooksley <bcooksley at kde.org> wrote:
> > > >
> > > > On Mon, Dec 16, 2019 at 5:21 AM Aleix Pol <aleixpol at kde.org> wrote:
> > > > >
> > > > > On Sat, Dec 14, 2019 at 1:08 AM Ben Cooksley <bcooksley at kde.org> wrote:
> > > > > >
> > > > > > On Sat, Dec 14, 2019 at 12:29 AM Nicolas Fella <nicolas.fella at gmx.de> wrote:
> > > > > > >
> > > > > > > I think Ben's concern is the load on our servers that is created by rebuilding each framework for each application build.
> > > > > >
> > > > > > Both. Ideally we don't want the builds to take any longer than they
> > > > > > really should, while at the same time maximising the scalability of
> > > > > > the system to handle all our builds (so we don't end up needing a
> > > > > > large farm of machines to support a comparatively small number of
> > > > > > builds).
> > > > > >
> > > > > > We utilise this approach with Krita's dependencies on Windows, MacOS
> > > > > > and Linux Appimages, where all the dependencies are built in a
> > > > > > separate set of dependency jobs and then reused by the application
> > > > > > build jobs.
> > > > > >
> > > > > > Obviously we wouldn't have a dependency job for every single Android
> > > > > > one, but a single shared one for all of them (being one per
> > > > > > architecture and build type - release/debug) should be more than
> > > > > > sufficient to cut down on the amount of stuff being built.
> > > > > >
> > > > > > >
> > > > > > > Nico
> > > > > >
> > > > > > Cheers,
> > > > > > Ben
> > > > > >
> > > > > > >
> > > > > > > On 13 December 2019 11:03:06 CET, Aleix Pol <aleixpol at kde.org> wrote:
> > > > > > >>
> > > > > > >> On Thu, Dec 12, 2019 at 7:35 PM Nicolas Fella <nicolas.fella at gmx.de> wrote:
> > > > > > >>>
> > > > > > >>>
> > > > > > >>>  Hi,
> > > > > > >>>
> > > > > > >>>  this is a breakout of https://mail.kde.org/pipermail/release-team/2019-December/011702.html since it's not really relevant for the release team.
> > > > > > >>>
> > > > > > >>>  The Android binary factory currently serves two purpose. It serves as a CI
> > > > > > >>>  system in the sense that it catches (compiletime) breakages and it provides
> > > > > > >>>  nightly APKs for beta users.
> > > > > > >>>
> > > > > > >>>  Given that we already have a few applications on Google Play and it's likely
> > > > > > >>>  that more will join in the future I would like to expand the scope of it to
> > > > > > >>>  also provide release builds from it, i.e. from stable branches, with binaries
> > > > > > >>>  built in release mode etc. This aims to replace the custom solutions some
> > > > > > >>>  projects have built for themselves.
> > > > > > >>>
> > > > > > >>>  On Donnerstag, 12. Dezember 2019 18:54:28 CET Ben Cooksley wrote:
> > > > > > >>>>
> > > > > > >>>>  With the way Android builds are currently conducted (building every
> > > > > > >>>>  single dependency from scratch each time) i'm opposed to doing this as
> > > > > > >>>>  it would double what is already quite an expensive load on the system
> > > > > > >>>>  already.
> > > > > > >>>>
> > > > > > >>>>  I'd like to see the Android build process optimised to let it reuse
> > > > > > >>>>  the results for most dependencies (Frameworks come to mind in
> > > > > > >>>>  particular) before we head in that direction.
> > > > > > >>>
> > > > > > >>>
> > > > > > >>>  Assuming that the general approach of using Docker for the builds is fine I
> > > > > > >>>  would like to propose this:
> > > > > > >>>  We provide a base image that contains the most common depenencies prebuilt,
> > > > > > >>>  i.e. Qt and the main Frameworks. We provide this in two variants, stable and
> > > > > > >>>  unstable. Stable would provide the latest frameworks release and would thus be
> > > > > > >>>  regularly rebuilt once a month after the Frameworks release. The unstable
> > > > > > >>>  version would contain a master Frameworks build to keep the CI/nightly aspect
> > > > > > >>>  of the builds. This would be rebuilt more often, i.e. once a day. This would
> > > > > > >>>  require a full Frameworks build per day which is still an improvement over 20
> > > > > > >>>  full Frameworks builds per day as we have now. With regard to the Qt version I
> > > > > > >>>  think sticking to the latest release as we do now is fine for both. A solution
> > > > > > >>>  that would not need a full Qt build once a day would thus be preferrable, e.g.
> > > > > > >>>  by using prebuilt Qt binaries or factoring out the Qt build into another base
> > > > > > >>>  image. Application builds then would pick an appropriate base image, build the
> > > > > > >>>  non-framework dependencies and finally the application itself. This should
> > > > > > >>>  reduce the non-necessary frameworks rebuilds by a significant amount.
> > > > > > >>>
> > > > > > >>>  @Ben do you agree with this proposal?
> > > > > > >>
> > > > > > >>
> > > > > > >> Am I right in the understanding that the reason to do this is we want
> > > > > > >> faster builds on binary-factory?
> > > > > > >>
> > > > > > >> Aleix
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Sent from my Android device with K-9 Mail. Please excuse my brevity.
> > > > >
> > > > > Hi,
> > > >
> > > > Hi Aleix,
> > > >
> > > > > I put this together:
> > > > > https://invent.kde.org/sysadmin/ci-tooling/commit/b1e5400926b44f59c56366c74317bb469eb703f3
> > > > >
> > > > > Could you please set it up to be run on build.kde.org? I haven't had
> > > > > time to do a proper testing round because I'm travelling today but it
> > > > > should just work.
> > > >
> > > > I've now setup https://build.kde.org/job/Administration/job/Docker%20Generate%20AndroidQt5.13%20AArch64%20KF5-SDK/
> > > >
> > > > I assume we would change the CI images to pull from this image rather
> > > > than the current main SDK image if this works out okay?
> > > > (Alternatively, we could fold the KF5 pre-compiled binaries into the
> > > > CI image if we don't think others would want to use it)
> > > >
> > > > >
> > > > > Aleix
> > > >
> > > > Cheers,
> > > > Ben