Future of Android binary factory builds

[Ben Cooksley]

On Sat, Dec 14, 2019 at 12:29 AM Nicolas Fella <nicolas.fella at gmx.de> wrote:
>
> I think Ben's concern is the load on our servers that is created by rebuilding each framework for each application build.

Both. Ideally we don't want the builds to take any longer than they
really should, while at the same time maximising the scalability of
the system to handle all our builds (so we don't end up needing a
large farm of machines to support a comparatively small number of
builds).

We utilise this approach with Krita's dependencies on Windows, MacOS
and Linux Appimages, where all the dependencies are built in a
separate set of dependency jobs and then reused by the application
build jobs.

Obviously we wouldn't have a dependency job for every single Android
one, but a single shared one for all of them (being one per
architecture and build type - release/debug) should be more than
sufficient to cut down on the amount of stuff being built.

>
> Nico

Cheers,
Ben

>
> On 13 December 2019 11:03:06 CET, Aleix Pol <aleixpol at kde.org> wrote:
>>
>> On Thu, Dec 12, 2019 at 7:35 PM Nicolas Fella <nicolas.fella at gmx.de> wrote:
>>>
>>>
>>>  Hi,
>>>
>>>  this is a breakout of https://mail.kde.org/pipermail/release-team/2019-December/011702.html since it's not really relevant for the release team.
>>>
>>>  The Android binary factory currently serves two purpose. It serves as a CI
>>>  system in the sense that it catches (compiletime) breakages and it provides
>>>  nightly APKs for beta users.
>>>
>>>  Given that we already have a few applications on Google Play and it's likely
>>>  that more will join in the future I would like to expand the scope of it to
>>>  also provide release builds from it, i.e. from stable branches, with binaries
>>>  built in release mode etc. This aims to replace the custom solutions some
>>>  projects have built for themselves.
>>>
>>>  On Donnerstag, 12. Dezember 2019 18:54:28 CET Ben Cooksley wrote:
>>>>
>>>>  With the way Android builds are currently conducted (building every
>>>>  single dependency from scratch each time) i'm opposed to doing this as
>>>>  it would double what is already quite an expensive load on the system
>>>>  already.
>>>>
>>>>  I'd like to see the Android build process optimised to let it reuse
>>>>  the results for most dependencies (Frameworks come to mind in
>>>>  particular) before we head in that direction.
>>>
>>>
>>>  Assuming that the general approach of using Docker for the builds is fine I
>>>  would like to propose this:
>>>  We provide a base image that contains the most common depenencies prebuilt,
>>>  i.e. Qt and the main Frameworks. We provide this in two variants, stable and
>>>  unstable. Stable would provide the latest frameworks release and would thus be
>>>  regularly rebuilt once a month after the Frameworks release. The unstable
>>>  version would contain a master Frameworks build to keep the CI/nightly aspect
>>>  of the builds. This would be rebuilt more often, i.e. once a day. This would
>>>  require a full Frameworks build per day which is still an improvement over 20
>>>  full Frameworks builds per day as we have now. With regard to the Qt version I
>>>  think sticking to the latest release as we do now is fine for both. A solution
>>>  that would not need a full Qt build once a day would thus be preferrable, e.g.
>>>  by using prebuilt Qt binaries or factoring out the Qt build into another base
>>>  image. Application builds then would pick an appropriate base image, build the
>>>  non-framework dependencies and finally the application itself. This should
>>>  reduce the non-necessary frameworks rebuilds by a significant amount.
>>>
>>>  @Ben do you agree with this proposal?
>>
>>
>> Am I right in the understanding that the reason to do this is we want
>> faster builds on binary-factory?
>>
>> Aleix
>
>
> --
> Sent from my Android device with K-9 Mail. Please excuse my brevity.