[Kget] Re: KDE/kdenetwork/kget/extensions/konqueror

[Dawit A]

On Mon, Nov 8, 2010 at 12:19 PM, Lukas Appelhans <l.appelhans at gmx.de> wrote:
> Am Montag 08 November 2010, 17:46:51 schrieben Sie:
>> On Mon, Nov 8, 2010 at 8:30 AM, Lukas Appelhans <l.appelhans at gmx.de> wrote:
>> > Hey!
>> >
>> > I just had time to test your changes and there is a problem with it I
>> > think...
>> >
>> > The download selected links option doesn't work with KHtml anymore... is
>> > this normal and is there a fix on the way?
>>
>> It is a TODO that needs to be implemented at the web browser level.
>> See kdelibs/khtml/khtml_ext.cpp lines # 1114 & 1152. The previous
>> implementation from KGet was simply not reused because it is very very
>> inefficient ;  the selected text gets reparsed again in order to
>> obtain the links. Anyhow, if a better approach cannot be found to
>> handle this, which is on my TODO list, then I will use the previous
>> implementation before the deep freeze for KDE 4.6...
> Thanks, those are the news I wanted to hear :)

This is now fixed and should work as well as it did before, but in the
process I found out about several issues in kget that I wanted to
inquire about...

#1. While re-implementing the "List selected links" capability of
kget's konqueror extension using the new KParts::HtmlExtension class,
I run into a problem where some of the selected links were not showing
up in kget's import links dialog. At first I thought there was some
issue with my implementation, but after some time spent debugging I
found out that kget purposefully discards what it deems as "web
content" (php, xml, html, asp pages) links by default in
KGetSortFilterProxyModel::filterAcceptsRow. Anyhow, why is that option
turned enabled by default ? Why is it not implemented as "Hide web
content" instead ? I am sure it is a sticky option that keeps its
state across app restarts ; so it would be retain its state. It sure
did confuse me and I have no doubt it will confuse others as well...

#2. kget makes an expensive call, KMimeType::findByUrl, for every link
it receive for listing and consequently downloading. That happens
because it does not provide a mechanism that allows callers to provide
it with content-type information about the links to be retrieved. If
such mechanism was available, then kget can completely avoid costly
link content-type look ups, for most links at least.

The good news is that it is relatively easy to add this feature into
the current dbus call, importLinks, without affecting the previous
behavior. Instead of sending a list of links only, we can send a list
links and content type separated by keywords and space, url <url> type
<content-type>. Here is an example that format:

   "url ftp://ftp.kde.org/pub/kde/stable/4.5.3/src/kdelibs-4.5.3.tar.bz2
type application/x-bzip-compressed-tar".

If the link does not contain any of these keywords and space, then it
will simply handled as it is being handled now. This would make it
possible to use KMimeType::mimeType(...) which avoids the unnecessary
network round trip for each and every link shown in the import dialog,
especially for remote protocols that support listing (ftp/sftp etc).
Best of all though, you can easily extend this format in the future to
add more information as such as "filesize". Anyhow, i have attached a
patch that implements that amongst other things.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: kget_import_links_improvement.patch
Type: application/octet-stream
Size: 5851 bytes
Desc: not available
Url : http://mail.kde.org/pipermail/kget/attachments/20101109/a384e649/attachment.dll