robots.txt in quickgit.kde.org

[Ben Cooksley]

On Mon, Dec 28, 2015 at 12:15 AM, Lydia Pintscher <lydia at kde.org> wrote:
> On Sun, Dec 27, 2015 at 12:08 PM, Ben Cooksley <bcooksley at kde.org> wrote:
>> On Sun, Dec 27, 2015 at 11:53 PM, Ashish Bansal
>> <bansal.ashish096 at gmail.com> wrote:
>>> Hi everyone,
>>
>> Hi Ashish,
>>
>>>
>>> "quickgit.kde.org" contains robots.txt[0] which is disallowing search
>>> engines to fetch the project repos. I just wanted to know if this is
>>> intentional or not?
>>>
>>> If I recall correctly, mirror of kde repositories on github was created just
>>> because it wasn't being indexed by the search engines.
>>
>> This is intentional, and is done to reduce the server load created by
>> indexers such as Google on the system hosting quickgit.kde.org.
>> (Generation of the pages, including the main index is substantially
>> more expensive than it appears due to the disk access required by
>> Git/SVN to return the needed information).
>
> Is there some place where search engines can easily index our source
> code or are we shooting ourselves in the foot here?

We could probably make it available by publishing the source trees
used by LXR / EBN.
This would only have the main branches obviously rather than everything though.

I haven't checked, but LXR may already make it's copy of the code accessible...

>
>
> Cheers
> Lydia

Regards,
Ben

>
> --
> Lydia Pintscher - http://about.me/lydia.pintscher
> KDE e.V. Board of Directors / KDE Community Working Group
> http://kde.org - http://open-advice.org
>
>>> Visit http://mail.kde.org/mailman/listinfo/kde-devel#unsub to unsubscribe <<