KIO: Mass Copy of Files from Different Sources to Different Destinations

[David Faure]

On Tuesday 08 September 2009, Dawit A. wrote:
> > It can lead to deadlocks when a high-level job needs multiple subjobs;
> > for instance KIO::file_copy() needs a Get and a Put job, and if only one
> > of these gets scheduled and not the other one, it will take a "slot" in
> > the scheduler for an indefinite amount of time; if another file_copy does
> >  the same, those two jobs could just wait forever on each other.
> 
> Hmm... the question I have is why would the end that got scheduled stay in
> queue for an indefinite amount of time ?

Because a FileCopyJob needs two subjobs to do anything: a get and a put.
If the get() is scheduled, but not the put(), then filecopyjob will get one
chunk of data and then will wait forever until the put job accepts it,
if the put job isn't scheduled.

> In other words why would this be a deadlock case ? 

FileCopyJob1 has get()-over-FTP scheduled but not put()-over-SFTP (blocked by 
job2)
FileCopyJob2 has get()-over-SFTP scheduled
  (e.g. the get() was scheduled just before FileCopyJob1's put)
but not put()-over-FTP, because it's blocked by job1's put.

Both jobs need the other one to terminate before they can proceed -> deadlock.

> Would the scheduled job not be serviced as soon as an
>  ioslave becomes available ? 

You have to differenciate low-level and high-level jobs.
Only low-level jobs (SimpleJobs) are known by KIO::Scheduler and scheduled by 
it.

>  Can you give an example of how to trigger this
>  dead lock ? I suppose I can simply start copying files from remote
>  locations (sftp/ftp) until the max instances limit is reached, no ?

Maybe. I admit I didn't actually try, but it seems logical to me, with the
above reasoning. To get many filecopyjobs started, I recommend copying a whole 
directory of files. That gives time to start another directory copy while it's 
happening. Each file being copied will start a FileCopyJob.

-- 
David Faure, faure at kde.org, sponsored by Nokia to work on KDE,
Konqueror (http://www.konqueror.org), and KOffice (http://www.koffice.org).