Critical Denial of Service bugs in Discover

[Fabian Vogt]

Moin,

Am Sonntag, 6. Februar 2022, 19:27:11 CET schrieb Ben Cooksley:
> On Sun, Feb 6, 2022 at 1:07 PM Fabian Vogt <fabian at ritter-vogt.de> wrote:
> > The first URL is used by kfontinst.knsrc from plasma-workspace:
> > ProvidersUrl=https://distribute.kde.org/khotnewstuff/fonts-providers.xml
> >
> > The second URL is used by multiple knsrc files in my VM:
> > aurorae.knsrc:ProvidersUrl=https://download.kde.org/ocs/providers.xml
> > comic.knsrc:ProvidersUrl=https://download.kde.org/ocs/providers.xml
> > kwineffect.knsrc:ProvidersUrl=https://download.kde.org/ocs/providers.xml
> > kwinscripts.knsrc:ProvidersUrl=https://download.kde.org/ocs/providers.xml
> > kwinswitcher.knsrc:ProvidersUrl=https://download.kde.org/ocs/providers.xml
> > wallpaperplugin.knsrc:ProvidersUrl=
> > https://download.kde.org/ocs/providers.xml
> 
> This makes me incredibly sad. We had a push to eliminate all usage of the
> legacy download.kde.org endpoint many years ago...
> I have now resolved the majority of these - if distributions could please
> pick up those patches that would be appreciated.
> 
> Please note that I have now terminated the support on the server that was
> making these legacy endpoints work, so those patches are necessary to
> restore functionality.

Does this decrease the server load noticably?

On IRC you wrote that the primary offender is
"KNewStuff/5.86.0-plasma-discover-update/", can you make the endpoints return
an error for the top entries only? Then we'll get bug reports only for the
likely cause(s) of the high traffic instead of everyone.

What might help is to have a lightweight proxy in front of Apache to handle
those paths in a way that doesn't stress the system much. That should probably
be investigated independently of the client side, as this is entirely under our
control and has immediate effects.

It's also possible that the requests aren't actually caused by Discover at all,
but just something which imitates it in a DDoS attack. In that case we couldn't
do anything on the client-side anyway. I don't think this is very likely, but
until the issue was reproduced with disover it's a possibility.

> > > Given that Sysadmin has raised issues with this component and it's
> > > behaviour in the past, it appears that issues regarding the behaviour of
> > > the OCS componentry within Discover remain unresolved.
> > >
> > > Due to the level of distress this is causing our systems, I am therefore
> > > left with no other option other than to direct the Plasma Discover
> > > developers to create and release without delay patches for all versions
> > in
> > > support, as well as for all those currently present in any actively
> > > maintained distributions, that disable all OCS functionality in the
> > > Discover updater. Distributions are requested to treat these patches as
> > > security patches and to distribute them to users without delay.
> >
> > Emergency workarounds for distributions might be to either not ship the KNS
> > backend by not building kns-backend.so or deleting it afterwards, or
> > disabling
> > the discover notifier
> > (/etc/xdg/autostart/org.kde.discover.notifier.desktop)
> > completely.
> 
> I have now committed patches to Discover going back to Plasma/5.18 which
> disable the build of the KNS backend.
> If distributions could please pick them up and distribute them as I
> previously indicated that would be much appreciated.
> 
> https://invent.kde.org/plasma/discover/-/commit/f66df3531670592960167f5060feeed6d6c792be

IMO we need a more targeted approach there. The main offenders aren't running
the latest version, so likely won't get those updates that quickly either.
If we have more data and can pinpoint it a bit better that would at least help
to speed patch delivery up.

> Please note that I intend to investigate whether it is possible to serve
> corrupted files from the server side that cause Discover to crash to help
> alleviate the load being created by those clients.

Sounds like a good way to DoS bugzilla instead and cause bad PR, both up- and
downstream. On top of that, it's possible that a resulting crashloop causes an
even higher frequency of requests.

Cheers,
Fabian

> Current load being generated by this is:
> 
> <dt>789 requests/sec - 6.4 MB/second - 8.3 kB/request - 1.70113
> ms/request</dt>
> <dt>217 requests currently being processed, 183 idle workers</dt>
> 
> > Cheers,
> > Fabian
> >
> Thanks,
> Ben