KIO directory listing - CPU slows down SSD

[Mark Gaiser]

On Sun, May 11, 2014 at 10:22 PM, Frank Reininghaus
<frank78ac at googlemail.com> wrote:
> Hi Mark,
>
> 2014-05-11 21:57 GMT+02:00 Mark Gaiser:
>> Hi,
>>
>> I've been playing with KIO speed improvements for quite a while now
>> and found some very interesting issues with KIO in combination with my
>> SSD drive: "Samsung SSD 840 PRO Series".
>>
>> My testcase is one directory filled with 500.000 files to test
>> directory listing speed. Please don't comment on this big number. I'm
>> well aware that it's insane! However, it shows bottlenecks that are
>> there but don't become visible with small sized folders like 1000
>> entries.
>>
>> Some numbers. Listing a directory using just C++ and Qt (so QT_STATBUF,
>> QT_READDIR and QT_LSTAT -- those are just platform defines. Nothing
>> custom is done there)
>>
>> 500.000 files: ~700ms
>>
>> Executing the same test using KIO::listDir:
>>
>> 500.000 files: ~4.500ms
>
> just to be sure, did you test this with a release build of Qt5 and KF5
> (i.e., with optimizations enabled)?

Yes :)
>
> Your mail reminds me that finishing the UDSEntry improvements that I
> have been working on some time ago [1] to kio.git is still on my TODO
> list - the main thing that I wanted to do before submitting a new
> review request is to set up a separate build of all of Qt 5 and KF5 in
> release mode because this is the only way to get reliable benchmark
> results. I'll try to get this done during the next week. This should
> speed up creating, writing and reading UDSEntries a lot because it
> saves much of the overhead that is required for the internal QHash in
> UDSEntry, in particular the memory allocations for the individual hash
> nodes. Then we might have a better base for estimating what else could
> be done to speed up directory listings.

Yes, i hope you could do that.
But please be aware that i also have a big patch pending that speeds
up UDSEntry a lot. I do that by:
- Storing all default 8 fields in UDSEntry (as class members)
- Keeping the hash for the custom fields (but rarely used now)

That greatly speeds things up and cuts memory down to 1/4th of what
you see today.
It even speeds up the test above to about ~3.500ms. A nice win but
still nowhere near my goal.

I can send you the patch later if you want. It's still W.I.P. and
currently breaks a few unittests :)
>
> Cheers,
> Frank
>
> [1] https://git.reviewboard.kde.org/r/113355/
> _______________________________________________
> Kde-frameworks-devel mailing list
> Kde-frameworks-devel at kde.org
> https://mail.kde.org/mailman/listinfo/kde-frameworks-devel