Critical Denial of Service bugs in Discover

[Ben Cooksley]

On Sun, Feb 6, 2022 at 1:07 PM Fabian Vogt <fabian at ritter-vogt.de> wrote:

> Hi,
>
> Am Samstag, 5. Februar 2022, 22:16:28 CET schrieb Ben Cooksley:
> > Hi all,
> >
> > Over the past week or so Sysadmin has been dealing with an extremely high
> > volume of traffic directed towards both download.kde.org and
> > distribute.kde.org.
> >
> > This traffic volume is curious in so far that it is directed at two paths
> > specifically:
> > - distribute.kde.org/khotnewstuff/fonts-providers.xml
> > - download.kde.org/ocs/providers.xml
> >
> > The first path is an "internal only" host which we were redirecting a
> > legacy path to prior to the resource being relocated to cdn.kde.org. The
> > second path has been legacy for numerous years now (more than 5) and is
> > replaced by autoconfig.kde.org.
> > It is of extreme concern that these paths are still in use - especially
> the
> > ocs/providers.xml one.
> >
> >...
> >
> > This indicates that the bug lies solely within Plasma's Discover
> component
> > - more precisely it's updater.
> >
> > Examining the origin of these requests has indicated that some clients
> are
> > making requests to these paths well in excess of several times a minute
> > with a number of IP addresses appearing more 60 times in a 1 minute sized
> > sample window.
>
> FWICT, this is caused by plasma-discover-update, which is triggered by the
> DiscoverNotifier service if automatic updates are enabled in kcm_updates,
> updates are available and the system idle for >=15min.
>
> // If the system is untouched for 1 hour, trigger the unattened update
> using namespace std::chrono_literals;
>
> KIdleTime::instance()->addIdleTimeout(int(std::chrono::milliseconds(15min).count()));
>
> (I wonder whether there's a bug about calling addIdleTimeout more than
> once.
> It will then invoke triggerUpdate multiple times after 15min of idle.)
>

That may explain why we are seeing so many requests from some IPs and very
few from others.


>
> The Discover KNS backend creates instances for all available knsrc files,
> which on construction call KNSReviews::setProviderUrl with the URL defined
> in
> those files, triggering the requests.
>

That does not sound scalable, and would certainly explain why not too long
ago we found that the traffic received by autoconfig.kde.org had grown to
such an extent we had to shift it to being handled by a CDN.
At the time I chalked the problem up to increasing popularity of our
software that included KNS functionality.


>
> The first URL is used by kfontinst.knsrc from plasma-workspace:
> ProvidersUrl=https://distribute.kde.org/khotnewstuff/fonts-providers.xml
>
> The second URL is used by multiple knsrc files in my VM:
> aurorae.knsrc:ProvidersUrl=https://download.kde.org/ocs/providers.xml
> comic.knsrc:ProvidersUrl=https://download.kde.org/ocs/providers.xml
> kwineffect.knsrc:ProvidersUrl=https://download.kde.org/ocs/providers.xml
> kwinscripts.knsrc:ProvidersUrl=https://download.kde.org/ocs/providers.xml
> kwinswitcher.knsrc:ProvidersUrl=https://download.kde.org/ocs/providers.xml
> wallpaperplugin.knsrc:ProvidersUrl=
> https://download.kde.org/ocs/providers.xml


This makes me incredibly sad. We had a push to eliminate all usage of the
legacy download.kde.org endpoint many years ago...
I have now resolved the majority of these - if distributions could please
pick up those patches that would be appreciated.

Please note that I have now terminated the support on the server that was
making these legacy endpoints work, so those patches are necessary to
restore functionality.


>
> > Given that Sysadmin has raised issues with this component and it's
> > behaviour in the past, it appears that issues regarding the behaviour of
> > the OCS componentry within Discover remain unresolved.
> >
> > Due to the level of distress this is causing our systems, I am therefore
> > left with no other option other than to direct the Plasma Discover
> > developers to create and release without delay patches for all versions
> in
> > support, as well as for all those currently present in any actively
> > maintained distributions, that disable all OCS functionality in the
> > Discover updater. Distributions are requested to treat these patches as
> > security patches and to distribute them to users without delay.
>
> Emergency workarounds for distributions might be to either not ship the KNS
> backend by not building kns-backend.so or deleting it afterwards, or
> disabling
> the discover notifier
> (/etc/xdg/autostart/org.kde.discover.notifier.desktop)
> completely.
>

I have now committed patches to Discover going back to Plasma/5.18 which
disable the build of the KNS backend.
If distributions could please pick them up and distribute them as I
previously indicated that would be much appreciated.

https://invent.kde.org/plasma/discover/-/commit/f66df3531670592960167f5060feeed6d6c792be

Please note that I intend to investigate whether it is possible to serve
corrupted files from the server side that cause Discover to crash to help
alleviate the load being created by those clients.
Current load being generated by this is:

<dt>789 requests/sec - 6.4 MB/second - 8.3 kB/request - 1.70113
ms/request</dt>
<dt>217 requests currently being processed, 183 idle workers</dt>


>
> Cheers,
> Fabian
>
>
>
Thanks,
Ben
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.kde.org/pipermail/plasma-devel/attachments/20220207/cdcdc691/attachment.htm>