KNewStuff web caching

[Ben Cooksley]

On Sat, Nov 16, 2024 at 8:27 PM Hy Murveit <murveit at gmail.com> wrote:

> I have a question about web-caching related to KNewStuff.
>

Hi Hy,


>
> *Background*
> I am a KStars developer, and asked Jasem (KStars lead) to post some "new
> stuff" for me.
> He did (or perhaps asked someone else to) and now when one goes to
> https://cdn.kde.org/edu/kstars/main.xml one gets a list that includes my
> "new stuff".
>
> Actually the kstars.knsrc says ProvidersUrl=
> https://indilib.org/jdownloads/knewstuff/providers.xml
>

This file really should be on autoconfig.kde.org.


> but when you go there it returns
>
> <knewstuffproviders>
> <provider downloadurl="https://edu.kde.org/kstars/downloads/main.xml"
> nouploadurl="https://edu.kde.org/contrib/" icon="
> http://edu.kde.org/pics/projects/cr32-app-kstars.png">
> <title>KStars Data</title>
> </provider>
> <provider downloadurl="https://indilib.org/jdownloads/knewstuff/dso.xml"
> nouploadurl="https://edu.kde.org/contrib/" icon="
> http://edu.kde.org/pics/projects/cr32-app-kstars.png">
> <title>KStars Data</title>
> </provider>
> </knewstuffproviders>
>
>
> and https://edu.kde.org/kstars/downloads/main.xml redirects to
> https://cdn.kde.org/edu/kstars/main.xml
>

That provider file should be updated to refer to the new home of those
resources to minimise unnecessary traffic being sent to edu.kde.org (which
is not CDN backed).
We would have caught that had it been hosted on KDE infrastructure.


>
> Anyway, here's the new item:
>
> <stuff>
> <name>ImagingPlanner Catalog1</name>
> <type>ImagingPlanner</type>
> <author email="kstars-devel at kde.org">Hy Murveit</author>
> <licence>Free noncommercial ShareAlike use with credit. Creative Commons
> license. Credit for individual images in .csv files.</licence>
> <summary lang="en">Catalog 1 for the KStars ImagingPlanner tool.</summary>
> <version>1.0</version>
> <release>1</release>
> <releasedate>2024-11-06</releasedate>
> <downloadsize1>6700</downloadsize1>
> <preview lang="en">
> https://cdn.kde.org/edu/kstars/previews/imagingplanner-screenshot.jpg
> </preview>
> <payload lang="en">
> https://cdn.kde.org/edu/kstars/ImagingPlanner-catalog1.tar.gz</payload>
> <rating>5</rating>
> <downloads>0</downloads>
> </stuff>
>
>
> I successfully use my browser to download/see
> https://cdn.kde.org/edu/kstars/ImagingPlanner-catalog1.tar.gz
> or
> https://cdn.kde.org/edu/kstars/previews/imagingplanner-screenshot.jpg
> So, all seems cool when using browsers!
>
> *The problem*
> The problem is that I can't see the new content from within KStars' KF5
> implementation of KNewStuff. I just see the "stuff" that existed before.
> Other folks using KStars on other machines do see my new stuff. Since I can
> see it with my browsers, and others can see it from within KStars, I assume
> the problem is that somehow/somewhere in the KDE libraries my local machine
> is caching the older web pages so that when KNewStuff goes to fetch it, it
> only sees the cached old content.
>

> Is that possible? If so, how would I clear that web cache?  This happens
> both when I run KStars on my Mac laptop, and when I run it on an Ubuntu VM
> on my laptop.  It doesn't happen on browsers from those machines (but I did
> clear the browser's caches when I had this problem early on).
>

As a starting point you should probably use kdebugdialog5 to enable all the
relevant debug output and capture the necessary logs to ensure KStars as
built on your machine is actually fetching what you think it is trying to
fetch and isn't fetching some old archive from elsewhere. KStars seems to
have KNewStuff content scattered around both cdn.kde.org and files.kde.org
so there is a possibility you are referencing an old file.


>
> Also, do you think there's a caching configuration issue on the hosting
> web server for that page?
>

Our CDN network is aggressively cached as the CDN relies on the CDN PoPs
fetching files from our backend server on first request and then holding a
cached copy.
For those wondering why we use this approach - Edge storage is
prohibitively expensive and we have ~188GB of data on cdn.kde.org (although
that is comparatively small vs. the 895GB of download.kde.org and 729GB of
files.kde.org).


>
> Thanks,
> Hy
>

Cheers,
Ben
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.kde.org/pipermail/kde-frameworks-devel/attachments/20241116/56c9633e/attachment.htm>