Sysadmin Load Reduction: Subversion Infrastructure

[Ben Cooksley]

On Mon, Nov 11, 2019 at 3:58 AM Luigi Toscano <luigi.toscano at tiscali.it> wrote:
>
> Ben Cooksley ha scritto:
> > On Sun, Nov 10, 2019 at 11:19 AM Albert Astals Cid <aacid at kde.org> wrote:
> >>
> >> El dissabte, 9 de novembre de 2019, a les 21:22:16 CET, Ben Cooksley va escriure:
> >>> On Sun, Nov 10, 2019 at 8:37 AM Albert Astals Cid <aacid at kde.org> wrote:
> >>>>
> >>>> El dissabte, 9 de novembre de 2019, a les 19:27:07 CET, Ben Cooksley va escriure:
> >>>>> On Sun, Nov 10, 2019 at 7:18 AM Alexander Potashev <aspotashev at gmail.com> wrote:
> >>>>>>
> >>>>>> сб, 9 нояб. 2019 г. в 03:20, Ben Cooksley <bcooksley at kde.org>:
> >>>>>>> This would include the shutdown of WebSVN in particular, which when
> >>>>>>> coupled with the shutdown of our two CGit instances would also allow
> >>>>>>> for us to eliminate an entire virtual machine from our systems.
> >>>>>>
> >>>>>> Will there be any web interface for SVN after shutdown of WebSVN?
> >>>>>>
> >>>>>> Can we assume https://phabricator.kde.org/source/svn/ remains
> >>>>>> available during the next 10 years?
> >>>>>>
> >>>>>
> >>>>> Phabricator's browser will be retired as part of the shutdown of
> >>>>> Phabricator, which will take place once Gitlab has assumed
> >>>>> responsibility for code hosting and review, and the tasks have been
> >>>>> migrated from Phabricator.
> >>>>>
> >>>>> Should WebSVN be shutdown as well, then there would be no web
> >>>>> interface to our SVN repository.
> >>>>
> >>>> That's not acceptable.
> >>>
> >>> Mind explaining why?
> >>
> >> Because we use it in l10n.kde.org to link to po files.
> >
> > Mind detailing what those links are used for?
> >
> >>
> >>> Bear in mind that there is a cost both in terms of infrastructure, and
> >>> people time to maintain a service such as WebSVN.
> >>
> >> We have money, we don't have to shut down things we use because there is a cost.
> >
> > I wasn't referring to monetary cost there, I was referring to the flow
> > on effects (such as having to maintain the necessary components on the
> > master server to allow for the Subversion repository to be mirrored).
> >
> > Note also the "people time" component there.
>
> Sure, but please see my previous questions:
> - can we extend the space of rosetta? It already has a partial checkout, and
> 100 GB of free space (which can be kept down

We would need to ask the system hosters to provide this, as Rosetta is
a donated system (if memory serves, Rosetta is currently IOPS
constrainted, so using it to host WebSVN may over-burden the system)

> - if that's not enough, can we simply setup a machine which periodically sync
> from the svn repository? You are probably going to tell me that it does not
> work without server support, but from what I'm reading about svnsync, I don't
> think it should overload the server if executed every 30 minutes.
> Are we sure that we still need something on the master server? Let's try it first.

I'm afraid you cannot use svnsync with KDE's Subversion repository,
that utility hasn't been able to handle it for many, many years now
(it crashes if memory serves - [ade] hit this and documented it on his
blog back around the time it broke which was before the switch to Git
got underway)

We have custom tooling which uses rsync instead to mirror the repository.

Custom tooling is necessary because plain rsync cannot be used to
reliably mirror the repository - because it has 1.5 million commits in
it, which means over 3 million inodes on disk. Each of these would
have to be checked by plain usage of rsync, which takes a substantial
amount of time even on NVMe SSD storage - during which time the local
mirror of the repository may be inconsistent (rendering it unusable),
and which also generates a matching disk load on the master server,
reducing it's performance.

The RSync endpoint for the mirrors to contact is the component on the
master server that has to be maintained.

> - in case it's not clear, I'm volunteering to maintain that system.
>
> Ciao
> --
> Luigi

Cheers,
Ben