[Kget] KGet2 multithreaded plugin

[Dario Massarin]

On Wednesday 01 March 2006 06:15, you wrote:
> Hi,
>
> Thanks for replying & for your interest!
>
> Metalink is not a mirror search, but that could be integrated. Instead of
> needing to search, the links are already right there.
>
> A Metalink is one long link that contains all the ways to get a file.
>
> A Metalink looks like this:
>
> http://mirrors.isc.org/pub/openoffice/stable/2.0.1/OOo_2.0.1_Win32Intel_ins
>t
> all.exe#!md5!66bd00e43ff8b932c14140472c4b8cc6#!metalink!ftp://ftp.ussg.iu.e
>d
> u/pub/openoffice/stable/2.0.1/OOo_2.0.1_Win32Intel_install.exe#!metalink!ht
>t
> p://mirrors.ibiblio.org/pub/mirrors/openoffice/stable/2.0.1/OOo_2.0.1_Win32
>I ntel_install.exe#!metalink!
> ftp://ftp.kddlabs.co.jp/office/openoffice/stable/2.0.1/OOo_2.0.1_Win32Intel
>_
> install.exe#!bt!http://ftp.snt.utwente.nl/OOo_2.0.1_Win32Intel_install.exe.
>t orrent
>
> This is in the format: main link + MD5 checksum + alternate links (x 3) +
> BitTorrent/P2P link. Each part is separated by #!xxx!
>
> A user clicks this one link. Then, the client takes this (admittedly long
> and slightly scary) link & breaks it down into its parts. The client
> downloads segments from each location. Then, when the download is complete
> it compares the given checksum with the actual checksum to make sure there
> are no errors etc. All the logic is in the client (like, obey speed limit
> or stop using a mirror if it gives less than 2k/sec etc).
>
> For normal users this is much simpler than manually searching for mirrors
> and using them & also simplifies the checksum comparing process. It is also
> backwards compatible with IE, FF, & Opera...& probably is in Konqueror.
> Clients that do not recognize Metalinks just fall back & download whatever
> is before the first #. I've included a page of example links.
>
> There is also more detailed info in the PDF.
>
> I've also included the patch for Metalink in FlashGot. It takes the
> Metalink & splits it up, then writes all the links into a file that
> GetRight imports. If you have used GR, there is a graphic in the PDF that
> might demonstrate this idea better than my words.
>
> I hope I am communicating the idea well, sometimes it is hard because I'm
> too closely involved. (I could probably capture a video that shows it in
> action if the above doesn't make sense). I believe it is pretty simple and
> pretty useful. It makes using mirrors & checksums so easy that people don't
> know they're doing it.

Very cool.

There are several ways to implement this in kget:
1) The easy way: we don't consider the torrent links at all, but use only the 
http and ftp ones and grab them inside the multithreaded plugin. This would 
be really really simple to implement, but not as powerful as it could be..

Anthony: do the other download managers allow to download the same file with 
torrents along with ftp and http?

2) The difficult way: in order to use at the same time and in the same file 
different tecnologies like bittorrent, http, ftp or whatever, we have to:
	a) allow a transfer to have child transfers. In this case, for example, the 
metalink transfer could have more than one child transfer (a multithread 
transfer to handle all the http and ftp links, plus a bittorrent transfer to 
handle the bittorrent link)
	b) abstract the concept of file source, each one providing informations like 
speed, for example. This makes it possible to centrally handle the whole 
downloading process, switching to faster sources and so on.
	c) this is complex: we should extend the transfer interface in order to let 
the transfer share their concept of "chunk". This would make it possible for 
a transfer to cooperate with another without downloading the same data.

I think that the first implementation could be done in a little time. It's 
just a matter of handling the link directly from the multithread plugin.

The second one requires A LOT of changes to the kget architecture, and this 
means that I have to think carefully about it.. This requires some time..

So I propose to go for Solution 1, if Manolo agrees, until I find out some 
time to consider the second way.

To Manolo: Does the multithread plugin alread handle different url sources? I 
ask you this becouse I wouldn't like you to spend too much time in code that 
will be drop in a second time. For now it should be enough a rough 
implementation, without any logic for choosing or switching source. In the 
case of metalink transfers, just take the first n urls and use them. A best 
solution will be implemented later, with Solution 2.

Best regards,
	Dario