Review Request: Improve mimetype detection for wevdav:// links
Dawit Alemayehu
adawit at kde.org
Sun Dec 2 14:49:13 GMT 2012
> On Dec. 1, 2012, 7:07 p.m., Dawit Alemayehu wrote:
> > Why not simply use KIO::get instead of KIO::mimetype ? For the record, one should never use KIO::mimetype for HTTP based protocols since it results in a HEAD request. It is a well known issue that sending a HEAD request to webservers, which for the most part generate content dynamically, is a useless proposition because the mime type they return on such request will be wrong most of the time (HEAD will not result in the generation of the content and hence the default type will probably be returned). Hence, the rule about not using KIO::mimetype for HTTP or any of its extensions like webdav. Instead you should simply do a KIO::get request by following the instructions on how to put a request on hold if all you want to do is determine the real mimetype of the content. This has even the side benefit of being able to use the same connection if you then decide to pull down the content afterwards.
> >
> > Havig said that, I am unable to test your patch because I do not have access. The links you provided are protected by HTTP authorization.
>
> Sergey Borovkov wrote:
> Yes, those links need authorization (unfortunately I don't know any webdav:// links that do not require it). About using KIO::get - thanks for useful infrmation, I did not know about this rule about not using KIO::mimetype for HTTP (did not see it anywhere), still it's really inconvenient to create additional code path for some protocols. Anyway bug when KIO::mimetype() returns just empty string is really annoying :( From what I understand it should never do that.
>
> Btw, just a sample ouptput from webdav.yandex.ru when doing HEAD request ( user and password omitted:) ):
> curl -I https://webdav.yandex.ru
> HTTP/1.1 415 Unsupported Media Type
> Yandex-Uid: 161224872
> Server: MochiWeb/1.0
> Keep-Alive: timeout=300
> Date: Sat, 01 Dec 2012 19:48:57 GMT
> Content-Length: 0
>
>
> Dawit Alemayehu wrote:
> Personally I would not use KIO::mimetype for any protocol unless there is an actual reason, e.g. I am simply listing the contents of a server. Then the chances the contents of any particular resource being are needed is rather small. Otherwise, I would simply use KIO::get and the put ioslave on hold functionality described in its API documentation because at some point right afterwards I will be retreiving the content.
>
> As far as the response from the webserver, I personally do not see anything wrong with it. A server is not under any obligation to send the content-type if it actually does not understand the client's request. That seems to be the case here. The server responded with a 415. That means it did not support or understand the request. As such it correctly returned a header that does not include a content type. A client can only reasonably expect to receive a valid mimetype when a server responds with a 2xx status code.
>
> The only issue I see is that what you posted above is done the curl. Can you enable the debug areas for kio_http following the instructions outlined at the link below and see what you get with kio_http instead ? The debug areas of interest are 7103 and 7113. You might need to install the debug versions of kdelibs for your distro if you did not compile from source. Otherwise, make sure you compiled in "RelWithDebInfo" or "Debug" cmake modes.
>
> [1] http://techbase.kde.org/Development/Tutorials/Debugging/Debugging_IOSlaves#How_to_get_debug_output
>
>
> Sergey Borovkov wrote:
> Sure, here it is:
>
> kio_http(8575) HTTPProtocol::sendQuery: ============ Sending Header:
> kio_http(8575) HTTPProtocol::sendQuery: "HEAD /%D0%94%D0%BE%D0%B1%D1%80%D0%BE%20%D0%9F%D0%BE%D0%B6%D0%B0%D0%BB%D0%BE%D0%B2%D0%B0%D1%82%D1%8C.pdf HTTP/1.1"
> kio_http(8575) HTTPProtocol::sendQuery: "Host: webdav.yandex.ru"
> kio_http(8575) HTTPProtocol::sendQuery: "Connection: keep-alive"
> kio_http(8575) HTTPProtocol::sendQuery: "User-Agent: Mozilla/5.0 (X11; Linux x86_64) KHTML/4.9.2 (like Gecko) Konqueror/4.9"
> kio_http(8575) HTTPProtocol::sendQuery: "Accept: text/html, text/*;q=0.9, image/jpeg;q=0.9, image/png;q=0.9, image/*;q=0.9, */*;q=0.8"
> kio_http(8575) HTTPProtocol::sendQuery: "Accept-Encoding: gzip, deflate, x-gzip, x-deflate"
> kio_http(8575) HTTPProtocol::sendQuery: "Accept-Charset: utf-8,*;q=0.5"
> kio_http(8575) HTTPProtocol::sendQuery: "Accept-Language: en-US,en;q=0.9"
> kio_http(8575)/kio_http_debug HTTPProtocol::sendQuery: sent it!
> kio_http(8575)/kio_http_debug HTTPProtocol::readResponseHeader:
> kio_http(8575) HTTPProtocol::readResponseHeader: ============ Received Status Response:
> kio_http(8575) HTTPProtocol::readResponseHeader: "HTTP/1.1 401 Unauthorized"
> kio_http(8575)/kio_http_debug HTTPProtocol::readResponseHeader: wasAuthError= false isAuthError= true sameAuthError= false
> kio_http(8575)/kio_http_debug HTTPProtocol::readResponseHeader: -- full response:
> "HTTP/1.1 401 Unauthorized
> WWW-Authenticate: Basic realm="Yandex.Disk"
> Server: MochiWeb/1.0
> Keep-Alive: timeout=300
> Date: Sun, 02 Dec 2012 07:33:56 GMT
> Content-Length: 35"
> kio_http(8575)/kio_http_debug HTTPProtocol::handleAuthenticationHeader: Trying authentication scheme: "Basic"
> kio_http(8575)/kio (kioslave) KIO::KPasswdServer::checkAuthInfo: window-id= 0 url= KUrl("webdavs://webdav.yandex.ru/????? ??????????.pdf")
> kio_http(8575)/kio (kioslave) KIO::KPasswdServer::checkAuthInfo: username= "" password=[hidden]
> kio_http(8575)/kio_http_debug HTTPProtocol::handleAuthenticationHeader: isError= false needCredentials= true forceKeepAlive= false forceDisconnect= false
> kio_http(8575)/kio_http_debug HTTPProtocol::proceedUntilResponseHeader: Previous Response: 0
> kio_http(8575)/kio_http_debug HTTPProtocol::proceedUntilResponseHeader: Current Response: 401
> kio_http(8575)/kio_http_debug HTTPProtocol::httpClose: keepAlive = true
> kio_http(8575)/kio_http_debug HTTPProtocol::cacheFileClose:
> kio_http(8575)/kio_http_debug HTTPProtocol::httpClose: keep alive ( 120 )
> kio_http(8575)/kio_http_debug HTTPProtocol::mimetype: ""
> klook(8572)/kio (KIOJob) KIO::TransferJob::slotFinished: KUrl("webdavs://webdav.yandex.ru/????? ??????????.pdf")
> klook(8572)/kio (Scheduler) KIO::SchedulerPrivate::jobFinished: KIO::MimetypeJob(0x224ccb0) KIO::Slave(0x1dd5b20)
> klook(8572)/kio (Scheduler) KIO::SchedulerPrivate::doJob: KIO::SimpleJob(0x1dd1e70)
> klook(8572)/kio (Slave) KIO::Slave::kill: killing slave pid 8575 ( "webdavs://webdav.yandex.ru" )
>
Hmm... almost. I wanted to see the actual response to the HEAD request. Not the 401 Unauthorized response. That I can see for myself.
- Dawit
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://git.reviewboard.kde.org/r/107518/#review22888
-----------------------------------------------------------
On Nov. 29, 2012, 5:42 p.m., Sergey Borovkov wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> http://git.reviewboard.kde.org/r/107518/
> -----------------------------------------------------------
>
> (Updated Nov. 29, 2012, 5:42 p.m.)
>
>
> Review request for kdelibs.
>
>
> Description
> -------
>
> KIO::mimetype() when used on webdav:// url returned empty string on some servers, empty string on others. After some investigating I found out that mime-type is determined like for usual http:// links - i.e. making HEAD requests.
> But this is problematic - some servers do not return content type for webdav links (for example this behavior occurred on webdavs://webdav.yandex.ru). Even if server returns content-type it can be wrong - for directories HEAD request return text/xml (that happens because webdav directories are simply xml listings).
> I tried to fix this by essentially calling stat on webdav url and emitting mimetype when it's found.
>
>
> Diffs
> -----
>
> kioslave/http/http.h 4d0b029
> kioslave/http/http.cpp b7f8e8d
>
> Diff: http://git.reviewboard.kde.org/r/107518/diff/
>
>
> Testing
> -------
>
> Tested this on two webdav servers (webdavs://webdav.yandex.ru and webdavs://dav.2safe.com).
>
>
> Thanks,
>
> Sergey Borovkov
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.kde.org/pipermail/kde-core-devel/attachments/20121202/df3760ce/attachment.htm>
More information about the kde-core-devel
mailing list