F-Droid meta data generation

[Volker Krause]

On Sonntag, 25. Oktober 2020 16:07:13 CET Aleix Pol wrote:
> On Sun, Oct 25, 2020 at 2:28 PM Volker Krause <vkrause at kde.org> wrote:
> > Hi,
> > 
> > I've been looking into the F-Droid meta data generation a bit, now that I
> > am able to run this locally and make changes to it.
> > 
> > The current system with generating the meta data from the final APK on the
> > signing system is nice in that it is entirely transparent for the app, but
> > it does have a few limitations:
> > - it's happening late in the pipeline, which makes some information hard
> > to
> > access (such as the source code repository URL)
> 
> You might be right about that. The repository URL is something fiddly
> in any case because there's many approaches that don't build software
> from git (e.g. tarballs).
> 
> I would like to include the git repository url in the appstream
> metadata for many reasons (de-duplication, development tools
> integration, etc.). This is something I could push for.
> That said, it might not solve the problem entirely.

True, extensions to the appstream file could be a part of the solution as 
well.

> > - it's running on the signing system, which requires deploying updates
> > manually, and presumably doing things like screenshot downloads on there
> > isn't ideal either from a security POV either
> 
> Yes, this was something that changed recently because Ben preferred to
> do it this way. I agree it's not ideal and again it's something we can
> probably improve upon by doing it closer to the developer setup.
> 
> > - it's not able to make use of manually provided parts for the meta data
> > (e.g. the banner image which has no equivalent in appstream).
> 
> True.
> We can have custom fields in appstream too, not sure if that is useful
> elsewhere.
> 
> > - it's not accessible for F-Droid release builds, ie. builds running on
> > their infrastructure
> 
> We can split the code outside, wherever makes sense.
> 
> > A possible way to address this could be to move meta data generation from
> > repo generation to the build process, implemented as part of ECM. The
> > resulting meta data would then need to be transferred alongside the APKs
> > to the signing system for integration into the F-Droid repo, presumably
> > this can be done by the same mechanism as the APKs are transferred.
> > 
> > I think this can be done in a similarly transparent way, while now also
> > being able to implement merging existing meta data fragments from the
> > source dir. By moving to ECM this also becomes independent of our CI
> > tooling, which should help F-Droid release builds.
> > 
> > Thoughts?
> 
> Overall I think it's good to make this process more reliable. I wonder
> what's the right way, the line between powerful and overengineered can
> be thin.

So far I'm hoping it wont change much in terms of complexity: 
- factor out the meta data generation from the current repo generation script, 
and feed it with with the appstream file from the source.
- look for existing meta data folders in the source (like e.g. ktrip has), and 
give that preference.
- output goes to the build directory where Jenkins can pick it up, or a F-
Droid release build.

> ECM would be a fine place to put it. Also the AndroidExtras repository
> we keep talking about could be a place. Our toolchain already depends
> on Python so it shouldn't be entirely out of place.

Right, thought about the AndroidExtras option as well, but I need more time to 
actually get that started...

> When you say "the same mechanism" you mean passing the metadata side
> by side or including the metadata within the apk?

That referred to the Jenkins artifact mechanism used to transfer the APKs. 
Putting it in the APK would theoretically work as well, but I'm not a big fan 
of putting unnecessary stuff in there.

> > @Nico: would that cover what F-Droid builds need?
> > @Aleix: could you add a license/copyright to the generation script (or
> > provide me with one) please, preferably with an ECM-compatible license?
> 
> Sure, no problem.
> https://invent.kde.org/sysadmin/binary-factory-tooling/-/merge_requests/120

Thank you!
Volker