KIO::ListJob::entries emits twice for folders with 100+ entries. Why?

[Mark]

On Thu, Nov 15, 2012 at 6:26 PM, David Faure <faure at kde.org> wrote:
> On Thursday 15 November 2012 13:12:27 Mark wrote:
>> My bad, the second signal is only emitting the remainder of whatever
>> comes after the first signal.
>> However, i still have the question as to why it happens at all?
>
> So that dolphin doesn't keep the user waiting in front of a blank view for a
> long time when opening /usr/bin. Better emit the entries in multiple batches,
> for incremental rendering.
>
Sorry, but i have to dive in deep.

The actual listing (listDir) on /usr/bin is done very rapidly!
Actually, all listings are done very rapidly even on remote locations.
Listing isn't the issue. The thing that makes this "seemingly" slow is
all the stuff that happens "after" the listing is in. Like determining
the mime type, and sorting the massive list. That is what makes it
slow.

Also, funny that you mention dolphin. Since dolphin can't possibly
make use of this. Dolphin is sorting the entries thus needs all of
them to be in before it can even sort. And even "if" dolphin receives
two entries signals (which is the itemsAdded signal from KDirModel)
then it certainly won't speed things up, but rather slow things down
because it would have to do sorting twice. Once on a small set of data
(relatively fast) and once on the full set of data (dog slow).

As a last remark regarding your dolphin example. If you open up a
large folder in dolphin you will see that dolphin waits till it has
all the contents of it. Then it starts displaying those. So in other
terms, it doesn't even benefit!
You can try this out yourself if you want. Go to a console and make a
new folder. Then type:
touch {1..100000}.txt

Now open up your dolphin in that folder and be prepared to wait :)

>> The two signal stuff for entries only seems to occur when i set:
>> job->addMetaData("details", "0");
>> If i just comment that line out i have just 1 signal from entries. Why
>> is it working like that? I would expect it to behave the same with or
>> without addMetaData("details", "0") set.
>
> The buffering of entries happens in SlaveBase::listEntry and is based on the
> time measurements and estimates and complex logic ;)
> Anyway, this shouldn't matter to your program, whether entries is called it
> one go or in multiple batches. If it does, your program is broken.

My program is working just fine.

This is the important part. Read again what i said and what your telling me now.
If i ADD addMetaData("details", "0") i get:
1 signal with the first 100 entries
1 signal with the remainder

If i DON'T add that "addMetaData" line and test the code out on a
folder that certainly takes about 10 seconds to get all the entries
then i'm welcomed by:
1 signal with all entries

You see where this is going? Listing the same folder WITH the
"addMetaData" line is going a lot more rapid then WITHOUT the
"addMetaData" line. Something is very wrong in the time calculation.
The results i'm getting should not be possible if the timing was
correct. Specially considering that the call with "addMetaData" takes
noticeable longer to run yet i only observe one entries signal.

>
>> I personally would just like to get rid of the first signal since in my
>> case all entries are feeded to a model anyway.
>
> I see several issues with that sentence.
> 1) Models must be able to react to dynamic changes (e.g. the user creates a
> file), so once they do that, they can also react similarly to "here are 10
> entries" followed by "here are 10 more entries" (= inserting rows).
> 2) Why not just use KDirModel? It handles all this for you.
>

I am actually using both. I made a breadcrumb bar which uses listDir
for every "crumb". I don't need a KDirModel there. For the contents i
am using a KDirModel.

Again, please!! Keep this about the entries signal. My app is just
fine and only serves to illustrate a case. It's the entries signal i'm
interested about since it seems to have an issue as i already clearly
indicated in my first post.