KIO::ListJob::entries emits twice for folders with 100+ entries. Why?

[Mark]

On Fri, Nov 23, 2012 at 10:42 AM, David Faure <faure at kde.org> wrote:
> On Friday 23 November 2012 00:49:11 Mark wrote:
>> I'm hitting a bit of an issue here in debugging SlaveBase::listEntry..
>> KIO is not running in the same thread as i'm running which makes
>> debugging tricky.
>
> Err, KIO doesn't really support threads. You have to run KIO in the main
> thread. Well if you connect the job to slots in another thread and never call
> any KJob API from that thread, I guess it can work out ok, but only then.
>
>> However, when i use GDB and set any breakpoints in slavebase.cpp in
>> the SlaveBase::listEntry funtion it does break but in my main
>> function..
>
> SlaveBase is in the kioslave, not in your application. So you need to kill
> kio_file and you need to type "kdeinit4" in a terminal.
>
> Otherwise kio_file gets started via an old kdeinit4 which has the old libkio
> loaded (the one without your debug output).
>
> I recommend kDebug over gdb for this, especially for a method that is called
> 500 times.
>
>> What i can do is grab the logic out ant test it in a separate C++ project.
>
> Well yeah you already posted a test program iirc.
> Otherwise here's mine: http://www.davidfaure.fr/2012/kio-listdir.zip
>
> --
> David Faure, faure at kde.org, http://www.davidfaure.fr
> Working on KDE, in particular KDE Frameworks 5
>

That really helped! Thanks a lot.

So my findings thus far.
I think there are two wrong areas here.
1. SlaveBase::listEntry is really using wrong names and types to get
something done. For instance, d->totalSize is a KIO::filesize_t type
and is being used to measure a list count. It's in reality no
"filesize" at all! That just adds to the confusion. I'm not sure - yet
- if the function itself is flawed in terms of time diffs.

2. this is where things get really interesting! The actual
file_unix.cpp listdir function seems wrong to me. [1]
The function FileProtocol::listDir is basically having two distinct
directory listings.
1 for the case where details are 0 (if)
1 for the case where the details are not 0 (else)

Both cases use a different "tactic". If the details are 0 then a
folder is read, file by file, and a listEntry(entry, false); is done
for each file. So here you do have possible IO time.
Now if the details are _not_ 0 then the tactic is different. The
folder is read in it's entirety inside a list (line 397 of [1]). Till
that moment it hasn't done any calls to listEntry(...) yet! Rather
iterates through the list again and builds up the UDSEntry stuff on
line 430 (also [1]).

While the step for details = 0 is faster (visually) it does include IO
time and is therefore probably slower in time compared to details > 0.
The details > 0 is only using CPU time and is probably faster thus
that's probably why i'm not seeing a second signal in the "details >
0" case.

But what to do about it?
My preference is restructuring FileProtocol::listDir to one of the two
cases that it's doing now and adjust FileProtocol::createUDSEntry to
handle details = 0 properly.

What's your opinion here?
Note: These issues that i find never stay on the surface do they..
This one is diving very deep in KIO, the shortcuts one was diving very
deep in KDE internals and even Qt..

[1] http://quickgit.kde.org/?p=kdelibs.git&a=blob&h=2b47b7cb1d9a31871210e31bc916f0ae41893b63&hb=e51062f18c94dda2b96987df6d73eb46ecd007f7&f=kioslave%2Ffile%2Ffile_unix.cpp
from line 320