RFC: KIO::Scheduler changes... [PATCH included]

[Dawit A.]

On Sunday 30 August 2009 20:57:16 Maksim Orlovich wrote:
> > New patch that combines all the individual patches I posted before (to be
> > applied from kdelibs level) with the following additional changes:
>
> I have a question for you... With these patches, is it still possible to
> bypass the schedule if needed, or at the very least to move a job to the
> front of the scheduler queue? 

Yes, there is a way to bypass the default scheduled mode, but you have to use 
the "connection-oriented" mode of the scheduler. No limit checking is done 
there. I am also contemplating to add the couple of lines required to remove a 
job from scheduled queue if you call KIO::Scheduler::doJob(...) again after 
calling KIO::get/put/... 

However, this should not be necessary for most applications. What the sheduler 
does currently is not really efficient at all. It allows unlimited number of 
connections to be opened to a single host. I think I have to explained the 
ramifications of that in the previous email.

> This is rather important for khtml, as different types of resources have
> vastly different effect on page load time.

Right. I am completely aware of that since I am currently working on kdewebkit 
; so I use both khtml and kdewebkit to test the effects of these changes. So 
far everything seems to work fine here, but of course I welcome any input from 
whomever wants to try these patches and give feedback...

> Also, maxInstances in particular sounds extremely problematic for use with
> konqueror and multiple tabs. With just 12 you can easily starve loading.

Well actually that is effectively how I tested everything... Opening multiple 
tabs and loading sites that make lots of request. The rendering speed of the 
sites seem unaffected with or without this patch. Single/multiple does not seem 
to matter. And that can be easily explained away by the fact that most of the 
requests when loading a site are relatively quick (small images, script/css 
files, xml data...) and take very little time. That in turn makes the ioslave 
available to process the next request. And these limits can be set to whatever 
value we deem is sane default, e.g. 20, 30, 100 etc is all fine with me. The 
important thing is that they are honored by the scheduler and people can 
change them afterwards...

BTW, I want to make sure one thing is understood here since it took me a long 
time to understand it as well. Since the "maxInstances" property deterimes the 
number ioslaves that can be active at a given time per protocol, for the http 
ioslave this can be up to 60 (12 * 5) active ioslaves at a time. Why ?  
Because the kio_http ioslave handles http, https, webdav and webdavs as well 
as proxied ftp requests which are all handled as specificly distinct protocol 
by the scheduler...