General Infrastructure Maintainability: API and EBN

[Ben Cooksley]

On Mon, Nov 11, 2019 at 12:41 AM Albert Astals Cid <aacid at kde.org> wrote:
>
> El diumenge, 10 de novembre de 2019, a les 1:48:10 CET, Ben Cooksley va escriure:
> > On Sun, Nov 10, 2019 at 11:20 AM Albert Astals Cid <aacid at kde.org> wrote:
> > >
> > > El dissabte, 9 de novembre de 2019, a les 22:03:37 CET, Ben Cooksley va escriure:
> > > > On Sun, Nov 10, 2019 at 9:29 AM Alexander Potashev <aspotashev at gmail.com> wrote:
> > > > >
> > > > > сб, 9 нояб. 2019 г. в 21:50, Ben Cooksley <bcooksley at kde.org>:
> > > > > > > > Over the past number of years one of the tasks the Sysadmin team has
> > > > > > > > worked on has been improving the overall maintainability of our
> > > > > > > > systems, with a significant number of specialised cronjobs, exceptions
> > > > > > > > and hidden linkages being eliminated.
> > > > > > > >
> > > > > > > > That is with one great exception: api.kde.org and ebn.kde.org.
> > > > > > > >
> > > > > > > > Both of these are suffering from an extreme amount of digital bitrot
> > > > > > > > and special casing and in general are now in a condition where I
> > > > > > > > cannot say for certain whether it would be possible to replicate the
> > > > > > > > setup on a new system without us experiencing some degree of breakage
> > > > > > > > (some of which we may not discover until weeks/months afterwards).
> > > > > > > >
> > > > > > > > In addition, the current setup relies on an old-fashioned overnight
> > > > > > > > reprocessing of all repositories, which is inefficient and resource
> > > > > > > > expensive. A more modern approach would have the various projects api
> > > > > > > > documentation generated on a delayed cycle from relevant branches as
> > > > > > > > part of something like a CI job (but not part of the actual CI
> > > > > > > > workflow itself).
> > > > >
> > > > > Hi Ben,
> > > > >
> > > > > I can't discuss this topic before we understand what exactly is wrong
> > > > > with api.kde.org and ebn.kde.org and why they are hard to manage.
> > > > > Could you please describe the current situation (where to find source
> > > > > code, how many servers, etc) in new Phabricator tasks like you did for
> > > > > identity.kde.org in https://phabricator.kde.org/T8449 ?
> > > > >
> > > > > P.S.  Complicated legacy systems can be easy to replicate once you
> > > > > automate their deployment by using Docker containers and/or
> > > > > configuration management software like Ansible.
> > > >
> > > > The problem isn't just the complicated legacy nature of them, but also
> > > > how fragile and impossible to maintain they are.
> > > >
> > > > There are currently to my knowledge the following ways of generating
> > > > documentation within the system as it currently stands:
> > > > - Legacy KDE 4 era generation tooling, still in use for large parts of
> > > > KF5 based applications (which needs periodic fixes, see
> > > > https://cgit.kde.org/kdelibs.git/commit/?id=12a8bb503351b98869f722ba932f822e3c495883)
> > > > - DoxyQML, which is in part reliant on the above KDE 4 era generation tooling
> > > > - Specialist handling for 'cmakedocs' and 'mkdocs' based projects
> > > > - New era KApiDox tooling
> > > >
> > > > On top of all of this, the entire process can only be run as one
> > > > single monolithic piece, which makes debugging and investigating
> > > > faults difficult.
> > > >
> > > > To use an example of this, back in February 2018 we received a request
> > > > (Sysadmin ticket) from someone reporting that their project's API
> > > > documentation wasn't being generated.
> > > > Despite investigation by Allen and others, we were unable to resolve
> > > > the issue, and recommended that the project in question be switched
> > > > from the KDE 4 era tooling to the newer KApiDox tooling, as it wasn't
> > > > possible to identify the fault.
> > > >
> > > > Earlier this year there was a fault with API generation in general
> > > > which broke a number of projects due to encoding of filenames/folders
> > > > (fixed in https://cgit.kde.org/websites/quality-kde-org.git/commit/?id=f71c68bc80cce68aa3bcc6a51626c8f22b7c73c3).
> > >
> > > I volunteer to fix it.
> >
> > Hi Albert,
> >
> > Thanks for the offer of help.
> >
> > Please note though that this is much more than just fixing what we
> > currently have.
> > It is re-engineering what we currently have to ensure that it is
> > maintainable, and meets the broader needs of the community in the long
> > term.
>
> Yes, i understand that.
>
> >
> > What we have currently doesn't really meet those standards - and as I
> > mentioned earlier, is in some cases impossible to debug and therefore
> > fix as everything appears to be working properly but ends up not
> > working.
> >
> > It would also be nice if we could use this as an opportunity for
> > someone not yet deeply involved to get involved more closely.
>
> That's kind of impossible though, asking a total newbie to take care of re-engineering a "old system" is almost a recipe for 0% hands raised.

Which is why I broke it out into several small pieces that each have
an achievable goal, and which in the case of both Parts 1 and 2, can
be tested locally by someone with no need for any special access on
our side (they don't even need a web server setup, their local web
browser should be enough for the most part).

Replacing the whole of api.kde.org may itself look like a very
daunting project, however getting some tooling together to generate
the API Documentation as a local folder of HTML files for a given Git
repository is a much simpler and smaller task, and is something
someone new could definitely tackle.

We can then aid them with scaling it up (which is Part 3 of the
proposal document I wrote up), which is actually a relatively straight
forward exercise.

>
> Cheers,
>   Albert
>

Cheers,
Ben

> >
> > >
> > > Cheers,
> > >   Albert
> > >
> >
> > Cheers,
> > Ben
> >
> > > >
> > > > >
> > > > > --
> > > > > Alexander Potashev
> > > >
> > > > Cheers,
> > > > Ben
> > > >
> > >
> > >
> > >
> > >
> >
>
>
>
>