Reusing of KIO slave instances

[Dawit A.]

[snipped]
> Hi,
> 
> (I took the liberty to reorder the messages above to end the top-posting)
> I also have a patch sitting around that enforces per-host and per-protocol
> connection limits. I think that the current scheduler code leaves much to
>  be desired, namely using appropriate data structures and modelling instead
>  of linear list scans for just about everything. Most to all required
>  features and/or constraints can actually be implemented much more directly
>  with a different approach, mostly a QMap<int, Job*> with serial numbers
>  (per host), and ~3 extra members in KIO::JobPrivate. Another QMap<int,
>  PerHostData*> keeps jobs ordered between hosts.

I agree 100% with you here. There is a need to make the scheduler more efficient 
than it is now. I also saw some of the work you started doing there. That and 
the fact that I do not have much time to work on the scheduler in addition to 
kdewebkit is the reason why I opted for the easiest possible solution that 
solved many of the issues with the current version. That means no optimization 
or changes to deal with efficiency which you outlined above.

> Additionally the scheduler currently conflates limits enforcing with
>  priorities which is a bad idea: high priority jobs don't observe any
>  limits, low priority jobs scheduled with KIO::Scheduler::ScheduleJob() do
>  observe per-protocol limits, IIRC. Priorities can be added to the design I
>  described with virtually no effort by skewing serial numbers.

And the problem is exasperated by the fact that almost all of the job requests 
are handled as if they are "high priority" jobs by default! Only in very few 
instances and manual intervention by application developers through the direct 
use of KIO::Scheduler::scheduleJob(...) do jobs actually become "low 
priority".

And that is exactly what my patch goes after. By default all jobs are 
scheduled and limits are strictly enforced, both per host and per protocol. 
Essentially with my patch every job is a low priority job without any 
distinction unless the "Connection oriented" approach, 
getConnectedSlaves(...)/assignJobToSlave(...), is used...

> I'm slow to finish and commit the patch because I'm busy with my Diplom
>  thesis (comparable but somewhat more work than Master thesis) :/

I did not commit mine for a different reason. Lack of testing. I cannot 
possibly test all applications that might be impacted by this patch. And since 
it is such a central player in the KIO land I wanted to be on the side of 
caution when dealing with such big change. However, it will not for sure 
receive any testing unless I commit it! A perfect catch-22...

> So Dawit, if you have something to commit *right now* and you coordinate on
> new API with me I'd say go ahead. 

I absolutely have no desire to take this any further than I already have. :) I 
actually do not have anytime for it. The patch I provided only makes a single 
API change by adding a function to KProtocolInfo so that the 
maxInstancePerHost property can be accessed easily. Nothing more. All the 
remaining changes were to implementation, private and protocol files.

> The desktop files should be fixed better sooner than later and it will be
> interesting to have another implementation when investigating bugs. In the
> medium term I want to make the scheduler much better though; I've seen a lot
> of code in the KHTML loader that  amounts to workarounds for
> KIO::Scheduler's deficiencies, especially lack  of priority support and
> inefficiency with a large number of pending jobs as well as the tendency to
> spawn infinite ioslaves per host and per protocol.

Well my patch should help with the spawning of many ioslaves (undue resource 
usage) and a tiny bit in terms of performance that results from that, 
prevention of excessive resource utilization both in both client and server. 
However, it does not deal with changing the data structures and containers for 
improving performance...

Anyhow, it would be great if you can work that out and improve the scheduler. 
Low level, but critical stuff like this always suffers because it is not the 
"sexiest" thing to work on and requires a lot of hard work and much more 
testing than working on bling. It is however the first thing people notice when 
things break!