Opinions on KIO Slave side sorting? Possible GSoC project?

[Mark Gaiser]

On Thu, Jan 8, 2015 at 9:14 AM, David Faure <faure at kde.org> wrote:

> On Sunday 04 January 2015 21:37:21 Mark Gaiser wrote:
> > Hi,
> >
> > I was reading back an mailing list thread i started about half a year ago
> > [1]. Later on in that thread Aaron suggested a cool idea of "slave side"
> > sorting [2].
> >
> > At that time i found it cool, but didn't really see a need for it.
> Dolphin
> > is doing a fine job at sorting anyway, so why would we bother moving it
> to
> > the KIO side. Well, a good reason for moving it to the KIO side just
> became
> > apparent to me when looking at this RR [3]. Sorting is already quite
> > complex in Dolphin. Adding in QCollator while also using threads makes
> it a
> > bit more nasty since a QCollator object per thread is required (hence the
> > RR).
> >
> > Sure, the end result (and Dolphin is probably going to move there) is to
> > use QCollatorSortKey which will make sorting quite a bit faster [4] and
> > remove a QCollator requirement, but it won't be any cleaner. Quite the
> > opposite, It will be more complicated due to the bookkeeping required to
> > use QCollatorSortKey.
>
> Moving complexity elsewhere doesn't make things less complex overall,
> though
> :-)
>

True :(


>
> > All this sorting stuff is quite complicated and only available in
> Dolphin.
> > Other people would need to duplicate it's efforts for the same results.
> > That and considering that KIO is a framework that can be used by external
> > (non KDE) parties made me think that having slave side sorting might be a
> > very good thing to have. It would allow for performance optimizations
> that
> > everyone can use and allows every KIO user to easily add sorting. It
> would
> > completely remove a need for Dolphin to have sorting and would greatly
> > simplify code that uses KIO.
>
> This is an argument for doing sorting in KIO. Sure, why not.
>
> But I don't see how it could be done slave-side. When using
> KDirModel/KDirLister the result of the directory listing is put into
> KDirListerCache and then reused from any view that wants to list that dir.
> These views might have different sorting... What then?
>

Regarding KDirListerCache, i think that should stay as it is. The sorting
should not influence the cache in anyway, the cache should just be a raw
container of the actual data like it is now.
When you want to have a certain sorted view you simply get the cache back
for that folder with a proxy on top of it that re-arranges the indexes in
the sorted order.

You can compare this with any Q*Model class with a QSortFilterProxyModel on
top of that. The principle is the same, the implementation would just be
without models and directly on QVector/QList objects.
The extra memory overhead for sorting would just be one vector of ints.
Fairly cheap :)

The issue i see here is different processes. KIO::listDir is a process,
KDirListerCache lives in the client process. You want to do sorting in both
processes.
1. When the client initially asks a listing you want to do the sorting as
quick as possible to send back as little data as possible. So sorting on
the SlaveBase side. As long as not all data is known in the KDirListerCache
sorting would have to be done on the SlaveBase side. That would be the case
to change sort order _while_ data is dripping in.
2. When all data is fetched you don't want to go over a socket to request a
different sorting method so then you would need to have the same sorting
operations on the KIO client side with data fed from KDirListerCache.

I do think that KIO would need a new class for this that can handle a set
of predefined sorting and filtering operations if this entire "slave side
sorting" is to be considered for KIO.


> Sorting in the slaves is mixing "core" and "gui" issues, IMHO. It's not the
> slave's job to know what kind of sorting the view wants.
>

> In fact I see no advantage in implementing sorting on the slave side
> compared
> to sorting on the app side. If it's about not doing that in the app's main
> thread so it remains responsive, then threads are the alternative as you
> already noted.
>

Just to be clear, when i said "slave side sorting" i meant sorting in - for
instance - SlaveBase. Not in the actual slave plugins.
Where exactly, i don't know, but certainly before data is being send back
to the application that requested a KIO::listDir.
Sorry if i confused you here.

The advantage i see with doing this is allowing data to be visible for the
user as soon as data is available which will give the user a smoother and
faster experience. It allows for a mechanism of "show me the first 100
items of this massive folder in this particular sort order". Something that
isn't fully possible right now since the client would have to wait till all
data has arrived over the socket, then sort it, then present it. All data
would still have to be fetched on the slave side, but only X number of
entries would have to be send to the client allowing it to immediately
present the data. The rest of the data can then be fetched on a "need to
know" basis. Eg. when scrolling down for the next batch of 100 items. By
that time the slave would probably (depending on the folder size and sort
order) be done fetching all items and is ready to immediately send those
pre sorted batches to the client.

An added advantage is that the client could then just use KIO as streaming
API. Just like a youtube api or whatever streaming api. The client would
just have to implement "canFetchMode" and "fetchMode" if the Qt classes are
used. It should prevent the client from needing to implement complicated
threading and UI tricks to keep the UI smooth.


>
> I haven't looked into QCollator in details, but isn't there a way to make
> it
> threadsafe?
>

Don't know. Perhaps Aleix knows, but that's for another thread anyway :)
Or perhaps a Qt bug report.

>
> --
> David Faure, faure at kde.org, http://www.davidfaure.fr
> Working on KDE Frameworks 5
>
>
CC'ing the people who initially sparked this idea.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.kde.org/pipermail/kde-frameworks-devel/attachments/20150108/03a96a5b/attachment.html>