KDirModelV2, KDirListerV2 and UDSEntryV2 suggestions

[David Faure]

On Wednesday 09 January 2013 11:15:20 Mark wrote:
> A little more in depth questions for KDirLister and KFileItem. In my
> profiling KFileItem ends up high due to various reasons, but
> KDirLister is also a bit of a heavy resource hog due to it's default
> behavior of fetching all file information (thus at least 1 stat call
> per file) which severely slows down the dir listing process for large
> folders.

This stat call happens in kio_file though, not in the GUI process where 
KDirLister lives, right?
So I'm surprised that you see that when profiling...
or is there a nasty stat() in KDirLister somewhere?

> My idea is as follows here. By default fetch as little information as
> possible. So make KDirLister fetch the folder content using "details =
> 0" by default. Perhaps with an additional KDirLister function (or
> flag) to change it's behavior to fetch all info (like it works now).
> The magic should happen in KFileItem. Right now that class isn't lazy
> loaded at all thus you get as many KFileItem instances as you have
> entries in the folder your listing. What i want to do here is make it
> lazy loading. When the class constructs it should have the immediate
> availability of the current UDSEntry without making anything heavy.
> The model can then use this class just fine. When a KFileItem is
> constructed it should (in the background) fetch the file to get all
> the details. When that's done it should notify the model of the data
> change which in turn makes the model update it's data.

You realize that "fetching the additional info" means an async kio job, in the 
general case (non local files), right?

Surely we don't want to do this in KFileItem itself (value class, no QObject, 
no signals), but in KDirLister itself. In fact, we don't want a KIO::stat per 
file, but a full-details listing for everything in one go, otherwise there will 
be a lot of jobs and roundtrips to kioslaves involved. More complexity in 
KDirLister, though, isn't something I'm looking forward to.

Did you check the speed difference between details==0 and details==2?
For local files it's "just" one stat() call per file (yes that can be a lot of 
calls, but I'd like proof that this actually makes a difference compared to 
everything else). And for other protocols there's no difference, even (with FTP 
you have to list the directory, which gives you full details anyway), so the 
two-stage approach will only make it slower.

> Using this approach the model can show data a lot faster (it doesn't
> have to wait for the file details anymore) and will automatically
> update it's entry when KFileITem tells it to do so.The net result for
> the user should be a much faster (visually) file browsing experience
> which should be a lot less taxing on the CPU and memory.

This only works in an icon view with no details and no filtering.
If you ask for details under the icons (size, etc.) it will come in delayed,
and if you use a list view, the details columns will come in delayed; visually 
this might be ok, or arguable, don't really know. This is starting to be a 
topic for kfm-devel so that the dolphin developers can give their input as 
well...

For sure it will break or complicate filtering. It's more rare so I guess it's 
ok if it works delayed, but it still has to be taken into account.

Overall, I'm just not sure the speed gain vs complexity ratio is high enough.
We already identified "reading .desktop files and .directory files" as something 
that has to be done delayed, I would very much favor doing that first, it's 
clearly a much bigger performance issue for network-mounted paths. Martin 
Koller had started some time ago, but I'm not sure what the current status is; 
if he doesn't have time to finish maybe you can take over that :-)

-- 
David Faure, faure at kde.org, http://www.davidfaure.fr
Working on KDE, in particular KDE Frameworks 5