[kde-community] Official KDE mirror on github

[Ben Cooksley]

On Mon, Aug 17, 2015 at 7:06 PM, Martin Sandsmark
<martin.sandsmark at kde.org> wrote:
> Hi!
>
> On Mon, Aug 17, 2015 at 08:55:57AM +0200, Jos van den Oever wrote:
>> What part of the KDE infrastructures can be fixed to make the repositories
>> easier to find?
>
> https://quickgit.kde.org/robots.txt
> https://projects.kde.org/robots.txt
>
> I think the reason for this should be pretty obvious; a ton of crawlers
> indexing everything we have is going to add an immense load, we have a huge
> amount of projects and source that they'll try to crawl now and then.
>
> And part of the reason for this (I think) is that both gitphp and
> chiliproject aren't the most performant.
>
> I don't remember the reason we run gitphp in the first place, but replacing
> it is not a trivial task in any way, and our sysadmins already have a ton of
> other work to do. Just guessing, but if we switch to e. g. cgit I think the
> load should be more realistic to handle (I run cgit on my own server, and it
> is extremely efficient).

Chiliproject is simply not scalable at the level search engines hit it
and the number of Git repositories we have loaded in it. We have 825
mainline repositories at the moment.

In terms of GitPHP - I suspect it could handle it, blocking spiders
was likely either an upstream decision or a reflexive one on our part,
after the (essentially) denial of service attacks that WebSVN has
experienced over the years from spiders. WebSVN is still protected by
numerous aggressive rules to prevent it being abused by both spiders
and people attempting to hotlink to resources on it.

We actually switched away from CGit to GitPHP due to issues it was
having - can't recall the specifics but I think it was crashing or
otherwise struggling to cope with the number of repositories we were
throwing at it at the time.

>
> So, I think the answer to your question is a) fix or replace our web
> interface(s) for git, and b) remove the robots.txt.
>
> --
> Martin Sandsmark

Regards,
Ben

> _______________________________________________
> kde-community mailing list
> kde-community at kde.org
> https://mail.kde.org/mailman/listinfo/kde-community