KDE Qt Patch Collection - Tarballs

[Neal Gompa]

On Wed, Oct 12, 2022 at 2:12 PM Ben Cooksley <bcooksley at kde.org> wrote:
>
> On Thu, Oct 13, 2022 at 12:53 AM Neal Gompa <ngompa13 at gmail.com> wrote:
>>
>> On Wed, Oct 12, 2022 at 6:23 AM Ben Cooksley <bcooksley at kde.org> wrote:
>> >
>> > Hi all,
>> >
>> > It has come to my attention after noticing abnormal activity on our Gitlab instance this evening that at least one distribution is fetching tarball archives for the KDE Qt Patch Collection directly from Invent.
>> >
>> > This is not a workload that invent.kde.org is intended to support and therefore places abnormal load on the system - especially in the case of QtWebEngine due to it's size. Analysis of the logs indicates that this type of activity is being conducted by both an RPM based distribution, as well as by Alpine Linux (see https://git.alpinelinux.org/aports/tree/community/qt5-qtserialbus/APKBUILD?h=3.16-stable)
>> >
>> > If the above applies to your distribution please ensure that you use your own copies of the tarballs generated by GItlab and do not have your automated tooling fetch it from invent.kde.org directly.
>> >
>> > Log excerpts, with IP addresses removed:
>> >
>> > [12/Oct/2022:05:17:59 +0000] "GET /frameworks/karchive/-/archive/8653117dede663d1f20850da82518fff4b7f732d/karchive-8653117.tar.gz HTTP/1.1" 200 1035786 "-" "rpmdev-spectool"
>> > [12/Oct/2022:06:25:03 +0000] "GET /qt/qt/qtwebchannel/-/archive/fa8b07105b5e274daaa8adcc129fa4aa0447f9f7/qtwebchannel-fa8b07105b5e274daaa8adcc129fa4aa0447f9f7.tar.bz2 HTTP/1.1" 200 249616 "-" "Wget/1.21.2"
>> > [11/Oct/2022:15:51:41 +0000] "GET /qt/qt/qtwebengine/-/archive/v5.15.8-lts/qtwebengine-v5.15.8-lts.tar.bz2 HTTP/1.1" 200 3090478 "-" "Wget/1.21"
>> >
>>
>> At this time, Fedora does not package snapshots yet, though I believe
>> I know who is doing it based on the log snippet here.
>>
>> But I'm going to use this to make a point here: I asked for
>> tag-releases of the KDE Qt patch set earlier this year because the
>> current situation is a huge mess[1]. The advice I continue to receive
>> from KDE developers is to just fetch HEAD and ship that. Aside from
>> being absolutely nuts to keep track of, you're telling us that the KDE
>> GitLab instance can't handle it.
>
>
> Fetching HEAD, in the form of cloning of the Git repository itself, is something our system is well optimised for and can handle without issue.
> Serving a compressed tarball of the files in that repository however is orders of magnitude more expensive and is not a workflow for which it is optimised to handle, especially not for automated access.
>
> It has always been Sysadmin policy that automated access to the repository browser functionality of any of our code repositories is not permitted.
> This has covered /*checkout*/ urls on WebSVN back when svn.kde.org was the primary repository and has continued throughout all of our Git repository hosting as well (and continues with /raw/ and /archive/ endpoints on Gitlab)
>
> Actual Git clones should be used.
>

Most of us are not going to do that, since our tooling is optimized
around tarballs. When I say "fetching HEAD" I mean pulling from master
using the tarball generator URL.

And the actual Git clones are slow and painful too anyway. Speaking as
a GitLab admin myself, it's not hard to break GitLab with clones,
especially big ones. :)




--
真実はいつも一つ！/ Always, there's only one truth!