KDE Qt Patch Collection - Tarballs

[Ben Cooksley]

On Thu, Oct 13, 2022 at 7:24 AM Neal Gompa <ngompa13 at gmail.com> wrote:

> On Wed, Oct 12, 2022 at 2:12 PM Ben Cooksley <bcooksley at kde.org> wrote:
> >
> > On Thu, Oct 13, 2022 at 12:53 AM Neal Gompa <ngompa13 at gmail.com> wrote:
> >>
> >> On Wed, Oct 12, 2022 at 6:23 AM Ben Cooksley <bcooksley at kde.org> wrote:
> >> >
> >> > Hi all,
> >> >
> >> > It has come to my attention after noticing abnormal activity on our
> Gitlab instance this evening that at least one distribution is fetching
> tarball archives for the KDE Qt Patch Collection directly from Invent.
> >> >
> >> > This is not a workload that invent.kde.org is intended to support
> and therefore places abnormal load on the system - especially in the case
> of QtWebEngine due to it's size. Analysis of the logs indicates that this
> type of activity is being conducted by both an RPM based distribution, as
> well as by Alpine Linux (see
> https://git.alpinelinux.org/aports/tree/community/qt5-qtserialbus/APKBUILD?h=3.16-stable
> )
> >> >
> >> > If the above applies to your distribution please ensure that you use
> your own copies of the tarballs generated by GItlab and do not have your
> automated tooling fetch it from invent.kde.org directly.
> >> >
> >> > Log excerpts, with IP addresses removed:
> >> >
> >> > [12/Oct/2022:05:17:59 +0000] "GET
> /frameworks/karchive/-/archive/8653117dede663d1f20850da82518fff4b7f732d/karchive-8653117.tar.gz
> HTTP/1.1" 200 1035786 "-" "rpmdev-spectool"
> >> > [12/Oct/2022:06:25:03 +0000] "GET
> /qt/qt/qtwebchannel/-/archive/fa8b07105b5e274daaa8adcc129fa4aa0447f9f7/qtwebchannel-fa8b07105b5e274daaa8adcc129fa4aa0447f9f7.tar.bz2
> HTTP/1.1" 200 249616 "-" "Wget/1.21.2"
> >> > [11/Oct/2022:15:51:41 +0000] "GET
> /qt/qt/qtwebengine/-/archive/v5.15.8-lts/qtwebengine-v5.15.8-lts.tar.bz2
> HTTP/1.1" 200 3090478 "-" "Wget/1.21"
> >> >
> >>
> >> At this time, Fedora does not package snapshots yet, though I believe
> >> I know who is doing it based on the log snippet here.
> >>
> >> But I'm going to use this to make a point here: I asked for
> >> tag-releases of the KDE Qt patch set earlier this year because the
> >> current situation is a huge mess[1]. The advice I continue to receive
> >> from KDE developers is to just fetch HEAD and ship that. Aside from
> >> being absolutely nuts to keep track of, you're telling us that the KDE
> >> GitLab instance can't handle it.
> >
> >
> > Fetching HEAD, in the form of cloning of the Git repository itself, is
> something our system is well optimised for and can handle without issue.
> > Serving a compressed tarball of the files in that repository however is
> orders of magnitude more expensive and is not a workflow for which it is
> optimised to handle, especially not for automated access.
> >
> > It has always been Sysadmin policy that automated access to the
> repository browser functionality of any of our code repositories is not
> permitted.
> > This has covered /*checkout*/ urls on WebSVN back when svn.kde.org was
> the primary repository and has continued throughout all of our Git
> repository hosting as well (and continues with /raw/ and /archive/
> endpoints on Gitlab)
> >
> > Actual Git clones should be used.
> >
>
> Most of us are not going to do that, since our tooling is optimized
> around tarballs. When I say "fetching HEAD" I mean pulling from master
> using the tarball generator URL.


> And the actual Git clones are slow and painful too anyway. Speaking as
> a GitLab admin myself, it's not hard to break GitLab with clones,
> especially big ones. :)
>

There is quite a bit here to unpack...

To start with, I find it *highly* concerning that you are disregarding my
request concerning the use of these endpoints simply because it does not
fit with your tooling.
As mentioned earlier, to protect the long term sustainability of KDE.org
infrastructure usage of the /archive/ and /raw/ endpoints of Gitlab from
anything automated is strictly prohibited.

With respect to Git clones, I must disagree with you there. It takes under
a minute to clone most Qt repositories from invent.kde.org for me.
In my experience, we have little issue with cloning even with all of our CI
nodes being busy and developers running kdesrc-build - there weren't even
any issues during Akademy to my knowledge, and that would have been the
busiest time for the system.

 Regards,
Ben


>
>
>
> --
> 真実はいつも一つ！/ Always, there's only one truth!
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.kde.org/pipermail/distributions/attachments/20221013/75e6258d/attachment-0001.htm>