Critical Denial of Service bugs in Discover

[Ben Cooksley]

On Mon, Feb 7, 2022 at 9:54 AM Fabian Vogt <fabian at ritter-vogt.de> wrote:

> Moin,
>
> Am Sonntag, 6. Februar 2022, 19:27:11 CET schrieb Ben Cooksley:
> > On Sun, Feb 6, 2022 at 1:07 PM Fabian Vogt <fabian at ritter-vogt.de>
> wrote:
> > > The first URL is used by kfontinst.knsrc from plasma-workspace:
> > > ProvidersUrl=
> https://distribute.kde.org/khotnewstuff/fonts-providers.xml
> > >
> > > The second URL is used by multiple knsrc files in my VM:
> > > aurorae.knsrc:ProvidersUrl=https://download.kde.org/ocs/providers.xml
> > > comic.knsrc:ProvidersUrl=https://download.kde.org/ocs/providers.xml
> > > kwineffect.knsrc:ProvidersUrl=
> https://download.kde.org/ocs/providers.xml
> > > kwinscripts.knsrc:ProvidersUrl=
> https://download.kde.org/ocs/providers.xml
> > > kwinswitcher.knsrc:ProvidersUrl=
> https://download.kde.org/ocs/providers.xml
> > > wallpaperplugin.knsrc:ProvidersUrl=
> > > https://download.kde.org/ocs/providers.xml
> >
> > This makes me incredibly sad. We had a push to eliminate all usage of the
> > legacy download.kde.org endpoint many years ago...
> > I have now resolved the majority of these - if distributions could please
> > pick up those patches that would be appreciated.
> >
> > Please note that I have now terminated the support on the server that was
> > making these legacy endpoints work, so those patches are necessary to
> > restore functionality.
>
> Does this decrease the server load noticably?
>

Unfortunately it doesn't have much impact, although this is hard to measure
as the volume of requests is not even and fluctuates quite a bit.


>
> On IRC you wrote that the primary offender is
> "KNewStuff/5.86.0-plasma-discover-update/", can you make the endpoints
> return
> an error for the top entries only? Then we'll get bug reports only for the
> likely cause(s) of the high traffic instead of everyone.
>

That is within our abilities yes - do we know if returning a 404 is
sufficient to trigger Discover to pop up a message to the user or is this
not something it will do?


>
> What might help is to have a lightweight proxy in front of Apache to handle
> those paths in a way that doesn't stress the system much. That should
> probably
> be investigated independently of the client side, as this is entirely
> under our
> control and has immediate effects.
>

Unfortunately I suspect that the vast majority of the CPU load will be from
setup/teardown of SSL sessions rather than the overhead of Apache itself.
As all of these requests are https:// there likely isn't much help that a
lightweight proxy can do.

I suspect i'll need to look into a Cloud based load balancer of some
description. Per https://www.digitalocean.com/products/load-balancer it
seems that their load balancer nodes are rated at around 250 HTTPS
connections per second.
To put that in context...

<dt>535 requests/sec - 5.0 MB/second - 9.6 kB/request - 7.16541
ms/request</dt>
<dt>146 requests currently being processed, 161 idle workers</dt>

So we'd need approximately 3 DigitalOcean load balancer nodes (cost of $30
USD / month or $360 USD / year) to handle this at current load numbers.
Earlier they were higher so in practice we are probably talking 4-5 nodes.

Given we're talking a single Octa Core with HT processor here I think it is
doing extremely well....

Note that even this is not a guaranteed silver bullet.


>
> It's also possible that the requests aren't actually caused by Discover at
> all,
> but just something which imitates it in a DDoS attack. In that case we
> couldn't
> do anything on the client-side anyway. I don't think this is very likely,
> but
> until the issue was reproduced with disover it's a possibility.
>
> > > > Given that Sysadmin has raised issues with this component and it's
> > > > behaviour in the past, it appears that issues regarding the
> behaviour of
> > > > the OCS componentry within Discover remain unresolved.
> > > >
> > > > Due to the level of distress this is causing our systems, I am
> therefore
> > > > left with no other option other than to direct the Plasma Discover
> > > > developers to create and release without delay patches for all
> versions
> > > in
> > > > support, as well as for all those currently present in any actively
> > > > maintained distributions, that disable all OCS functionality in the
> > > > Discover updater. Distributions are requested to treat these patches
> as
> > > > security patches and to distribute them to users without delay.
> > >
> > > Emergency workarounds for distributions might be to either not ship
> the KNS
> > > backend by not building kns-backend.so or deleting it afterwards, or
> > > disabling
> > > the discover notifier
> > > (/etc/xdg/autostart/org.kde.discover.notifier.desktop)
> > > completely.
> >
> > I have now committed patches to Discover going back to Plasma/5.18 which
> > disable the build of the KNS backend.
> > If distributions could please pick them up and distribute them as I
> > previously indicated that would be much appreciated.
> >
> >
> https://invent.kde.org/plasma/discover/-/commit/f66df3531670592960167f5060feeed6d6c792be
>
> IMO we need a more targeted approach there. The main offenders aren't
> running
> the latest version, so likely won't get those updates that quickly either.
> If we have more data and can pinpoint it a bit better that would at least
> help
> to speed patch delivery up.
>

>From the data we are seeing, it appears that the vast majority of the
clients are running the exact same set of software.
This information comes from files.kde.org, which hosts a series of legacy
and static datasets for the KDE Education family of applications.

   9117 "KNewStuff/5.86.0-discover/5.22.5"
   9331 "KNewStuff/5.90.0-discover/5.23.5"
  52737 "Mozilla/5.0"
1852450 "KNewStuff/5.86.0-plasma-discover-update/"

This would indicate the distribution in question is shipping Frameworks
5.86.0 (which from my understanding is a small handful of distros -
including Ubuntu 21.10) alongside a version of Plasma including the updater.
Unfortunately we don't know more than that as the amount of information
available to us is very limited.

The only other data point we have is our server monitoring system:

[image: 80029574.png]

This indicates that system load has increased steadily from October onward,
which would fit with an increasing number of users updating to Ubuntu 21.10
and receiving a system with the affected versions - however this is not
particularly conclusive.


>
> > Please note that I intend to investigate whether it is possible to serve
> > corrupted files from the server side that cause Discover to crash to help
> > alleviate the load being created by those clients.
>
> Sounds like a good way to DoS bugzilla instead and cause bad PR, both up-
> and
> downstream. On top of that, it's possible that a resulting crashloop
> causes an
> even higher frequency of requests.
>
> Cheers,
> Fabian
>

Thanks,
Ben


>
> > Current load being generated by this is:
> >
> > <dt>789 requests/sec - 6.4 MB/second - 8.3 kB/request - 1.70113
> > ms/request</dt>
> > <dt>217 requests currently being processed, 183 idle workers</dt>
> >
> > > Cheers,
> > > Fabian
> > >
> > Thanks,
> > Ben
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.kde.org/pipermail/plasma-devel/attachments/20220207/996cc977/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 80029574.png
Type: image/png
Size: 119516 bytes
Desc: not available
URL: <http://mail.kde.org/pipermail/plasma-devel/attachments/20220207/996cc977/attachment-0001.png>