Using KIO to retrive HTTP Headers [GSoC student help request]

[Aish Raj Dahal]

On Sun, Jul 8, 2012 at 7:51 PM, Dawit A <adawit at kde.org> wrote:
>
>
> On Sun, Jul 8, 2012 at 4:29 AM, David Faure <faure at kde.org> wrote:
>>
>> On Thursday 28 June 2012 06:13:15 Aish Raj Dahal wrote:
>> > The issue is that KIO attempts to download the entire file before
>> > being able to query for its HTTP header responses. This behavior may
>> > be taken lightly for smaller files, however for larger files
>> > downloading an entire file, just to get the HTTP header response
>> > associated with it seems to be impractical.
>>
>> As Albert said, the headers are sent as soon as they are received, and the
>> mimetype signal is emitted before the contents of the file, this is by
>> contract
>> for KIO::get(). This is how we can put the job on hold while launching the
>> proper application for that url, and the app then resumes the download in
>> order to get the file contents.
>>
>>
>> Alternatively you could use KIO::mimetype which performs an HTTP HEAD
>> request
>> on the given URL, but we had the experience (many years ago) that some
>> webservers didn't reply correctly to HEAD requests.
>
>
> The other alternative (actually a hack) you have is to connect to the KIO's
> mimeType signal and put the ioslave on hold as outlined in KIO::get's
> documentation. That should allow you to retrieve the header before
> downloading the file. You can then resume the download by issuing a second
> get request.
>

Respected KDE developers,

As suggested above in both of the emails, I have tried using
KIO::mimetype as well as KIO's mimetype respectively to get the HTTP
headers. My effort in both the cases have completely been in vain.

While discussing the issue with dfaure on #kde-devel I came across
what may be actually a possible bug with the HTTP KIO slave.
The issue occurs when trying to get the HTTP headers after setting
setRedirectionHandlingEnabled to false.

Here are two different scenarios where the possible problem with KIO
can be observed

1) Case One : When mimetype signal emitted by KIO::TransferJob is used

In order to clarify more, let me take an example file
https://github.com/ardahal/kio-learner/blob/ard-dev/metalinkHttp/metalinkHttp.cpp
. The given file uses the mimetype signal (at line 44) to get the
headers as soon as the mimetype is emitted. The catch is, since we do
no want the redirected HTTP headers but instead want the original HTTP
headers, setRedirectionHandlingEnabled has been set to false. This
program when run, does not emit the mimetype signal as all, and as a
result the qDebugs at line 51 and 52 are never executed . This
behavior is seen not only for URLs which redirect (like
http://www.example.com ) but also for URLs which have no redirection
(like http://www.google.com.np) . On the other hand if
setRedirectionHandlingEnabled is set to true, the mimetype signal is
emitted , though the headers are not of the original server but of the
redirected ones (which is not intended).

2) Case Two :  When KIO::mimetype

For better clarification, here is the test file when I used this
approach: https://github.com/ardahal/kio-learner/blob/experimental/metalinkHttp/metalinkHttp.cpp
. Here, KIO::mimetype is used to make an HTTP HEAD request to the
server. In this case too like the previous case,
setRedirectionHandlingEnabled is enabled to false, and once again the
program fails to execute the content of the slot that is connected to
the result signal. In addition to this, even if
setRedirectionHandlingEnabled is set to true, the content under the
slot that is connected to the result signal (mimeResult in the
aforementioned link at line number 50) .

In addition to the above two approaches, performing a KIO::get on the
KUrl and connecting its result signal along with
setRedirectionHandlingEnabled although is able to get the headers when
the file that is referenced by KUrl is small, KIO attempts to download
the file before getting the headers when the file is large. I have a
working implementation of this approach here
https://github.com/ardahal/kio-learner/blob/master/showHttp/showHttp.cpp
.

Now, in case you would like to have a more closer look at the code
that I'm using to test these features, please refer to
https://github.com/ardahal/kio-learner (
git://github.com/ardahal/kio-learner.git ).
The branch "ard-dev" contains the implementation using  the mimetype
signal, while the branch "experimental" contains an effort to do the
same using KIO::mimetype.

I would be really glad if you could spare some time to look into this matter.

Thank You very much for taking time to read through this long email. :)

Best Regards,
Aish Raj Dahal
Google Summer of Code Student
Metalink / KGet