Retirement of SVN Commitfilter and Legacy Get Hot New Stuff systems

[Ben Cooksley]

On Sun, Jul 30, 2017 at 6:35 PM, Martin Flöser <mgraesslin at kde.org> wrote:
> Am 2017-07-29 22:18, schrieb Ben Cooksley:
>>
>> On Sun, Jul 30, 2017 at 3:33 AM, Olaf Schmidt-Wischhöfer
>> <olaf at amen-online.de> wrote:
>>>
>>> Ben Cooksley:
>>>>
>>>> I've checked and it appears that only a small handful of applications
>>>> still use newstuff.kde.org:
>>>> - KBlocks
>>>> - KDiamond
>>>> - KGoldRunner
>>>> - Kigo
>>>> - KSirk
>>>> - KSnakeDuel
>>>> - KSysguard
>>>>
>>>> These applications should all be ported to use store.kde.org.
>>>
>>>
>>> How long will it take for users to get the updates? And are we planning
>>> to ignore users on, for example, Debian?
>>
>>
>> That is up to distributions. Unfortunately we need to break things
>> from time to time on the server side.
>>
>> As I mentioned, the web server for newstuff.kde.org has been down for
>> several months, so this just makes the current arrangements permanent.
>>
>> If we had to wait until all users were able to receive an update to
>> change something like this we would never be able to make any changes
>> to our systems. To give an example, the original GetHotNewStuff
>> implementation as used by many KDE 3 and early KDE 4 applications
>> still receives a large number of requests.
>
>
> But that is a point for not breaking it! If we still have users, we
> shouldn't break it!

Then I can tell you that in some cases making any change would be
absolutely impossible.
I have to choose between maintainability of our infrastructure and
compatibility, and in this case maintainability wins out.

To give some examples... we would never have been able to upgrade to
Bugzilla 4 if your above requirement had to remain true. DrKonqi until
our conversion to Bugzilla 4 (which introduced the REST API) was
reliant on scraping the web pages. The content and layout of those
pages were changed by the upgrade.

We kept compatibility measures in place within Bugzilla 4 for a good 2
years or so to avoid that breakage. There is a cost to that
(modifications to the Bugzilla core code, along with tweaks to the
theme - all of which had to be merged on each update including
security updates).

As another example, moving resources fetched by applications has
caused us issues in the past. I can cause at least one application to
crash simply by removing one line from an Apache config. In another
case we had to put a cronjob in place to copy metadata files from one
system (with lots of disk space) to the old location which older
versions of an application reference to keep them working.

These changes have to be maintained, taken into account when making
changes to other systems, and carried over when we perform system
migrations and upgrades.

Both of those issues were due to developers using
QNetworkAccessManager, which by default, doesn't enable redirects. The
Qt Installer Framework uses QNetworkAccessManager to make HTTP
requests, and crashes whenever it gets hit with a redirect (at least
in it's Qt 4 variant).

The reason the legacy gethotnewstuff system still works is because it
uses static files (which have no maintenance cost) and it uses KIO on
the client side - which handles redirects correctly. The maintenance
cost to Sysadmin of it is basically nil.

What you're essentially asking here is for Sysadmin to takeover
maintenance responsibility for a set of scripts which we don't fully
understand, which have been hardcoded to use Postgres (a system which
isn't really used by us) and which may have other unknown
dependencies. That seems a bit unreasonable to me.

>
> Seriously that is one of the points where we can significantly differ from
> proprietary providers where random stuff breaks because services get shut
> down.
>
> Is there no way to redirect it in a way that it can continue to work?

Unfortunately no.

>From what I can tell, newstuff.kde.org speaks the OCS protocol, which
in applications is handled by the Attica library (through knewstuff).
Unfortunately Attica uses QNetworkAccessManager and redirect handling
was only added sometime last year, and only to the Qt 5 variant.

KWin 4.11 is forever stuck using a Qt 4 version of Attica, which can't
handle redirects. It might even crash if we try to do redirects. We
can't even do upgrades to HTTPS (which is enforced on all the servers
Sysadmin maintains with few exceptions).

>
> Cheers
> Martin

Regards,
Ben