KIO directory listing - CPU slows down SSD

[Frank Reininghaus]

Hi Mark,

2014-05-11 21:57 GMT+02:00 Mark Gaiser:
> Hi,
>
> I've been playing with KIO speed improvements for quite a while now
> and found some very interesting issues with KIO in combination with my
> SSD drive: "Samsung SSD 840 PRO Series".
>
> My testcase is one directory filled with 500.000 files to test
> directory listing speed. Please don't comment on this big number. I'm
> well aware that it's insane! However, it shows bottlenecks that are
> there but don't become visible with small sized folders like 1000
> entries.
>
> Some numbers. Listing a directory using just C++ and Qt (so QT_STATBUF,
> QT_READDIR and QT_LSTAT -- those are just platform defines. Nothing
> custom is done there)
>
> 500.000 files: ~700ms
>
> Executing the same test using KIO::listDir:
>
> 500.000 files: ~4.500ms

just to be sure, did you test this with a release build of Qt5 and KF5
(i.e., with optimizations enabled)?

Your mail reminds me that finishing the UDSEntry improvements that I
have been working on some time ago [1] to kio.git is still on my TODO
list - the main thing that I wanted to do before submitting a new
review request is to set up a separate build of all of Qt 5 and KF5 in
release mode because this is the only way to get reliable benchmark
results. I'll try to get this done during the next week. This should
speed up creating, writing and reading UDSEntries a lot because it
saves much of the overhead that is required for the internal QHash in
UDSEntry, in particular the memory allocations for the individual hash
nodes. Then we might have a better base for estimating what else could
be done to speed up directory listings.

Cheers,
Frank

[1] https://git.reviewboard.kde.org/r/113355/