KIO: Mass Copy of Files from Different Sources to Different Destinations

[Dawit A.]

On Thursday 20 August 2009 07:27:59 David Faure wrote:
> On Thursday 20 August 2009, Dawit A. wrote:
> > hmm... would this not mean that the "Direct" scheduling method that has
> > been the default will go away completely ? In other words, there is no
> > way to have a single job per single connection (read: ioslave) support
> > anymore ? I am sure most application will remain unaffected, but if there
> > are any applications out there that somehow relied on this behavior, then
> > they will be broken, no ?
>
> To be honest I don't really understand the benefit of "direct scheduling",
> or the difference it makes to the application.
> We always have a single job per kioslave, that's how kio works. At a given
> moment, a kioslave is only handling requests for one job anyway.

Well yes a kioslave only handles a request for one job, but the issue here is 
how many of those ioslaves should the scheduler create to service 
jobs/requests. "Direct scheduling" guarantees that if I make 4 requests one 
after the other, all those requests will immediately spawn a kioslave. That is 
I will get a new connection for each of my requests. The kioslaves are only 
reused if and when they become idle. In the scheduled case, no matter what 
everything is queued and processed serially one after the other.

> AFAICS the difference is only between
> "if no kioslave is idle, create another kioslave _now_ for this job"
> and
> "if no kioslave is idle, create a kioslave if possible (not too many
> concurrent slaves) and otherwise wait for reusing one when it's available"

Right, but that is a big difference though, "one connection per request" vs 
"one connection per any number of requests". The latter is much more 
restrictive...

> In all cases jobs are async anyway, so it's only about "not waiting much"
> (but risking a "too many connections to this FTP server") and "maybe
> waiting a bit", but then connection limits are honored.
>
> I don't think we'd be losing or breaking anything, but maybe I'm missing
> something.

Well the "not waiting much" is a very important point here. And as an example 
I will give you a download manager that provides the user the option to set 
the number of connections to make per site. How would it be able to do that if 
the default is changed to "Scheduled" and "Direct" is completely gone ? It 
will not break the application completely, but all the downloads will be done 
one at a time because the jobs are queued and have to wait on one another. In 
other words, I cannot guarantee the x number of connections and start those 
many downloads at the same time.

In addition to the fact that KIO::Scheduler::scheduleJob does not allow 
anything but a SimpleJob (dunno why that restriction) to be scheduled, the 
function name itself is the source of confusion. It would have been better if 
the function was named queueJobs or something along that line and allowed you 
to provide the maximum number of ioslaves that can be spawned to service any 
request per site/location. That should solve many issues like this one. Even 
in KIO::AccessManager that would have been a much better approach. However, 
even then the default of "Direct" should not be changed IMHO...