KIO directory listing - CPU slows down SSD

[Mark Gaiser]

On Mon, Jun 2, 2014 at 6:42 PM, Aaron J. Seigo <aseigo at kde.org> wrote:
> On Thursday, May 29, 2014 16:32:28 Mark Gaiser wrote:
>> On Thu, May 29, 2014 at 12:21 AM, Aaron J. Seigo <ase
>
>> It's just a hobby for me to optimize it to levels that match raw speed
>> (without any KIO).
>
> meaningful speed improvements would be a reason to change things.
>
> if wire compatibility can be maintained while reaping speed benefits (and i
> think we've seen it can be done ...), then that isn't a reason.
>
>> I'm "trying" to improve things by making them simpler. If that
>> requires a new code path: fine with me.
>
> simplicity is a good goal, too. personally, i'd count it as a minor goal when
> looking at a long-maintained/debugged codebase, though. disturbing too much in
> the quest for simplicity often ends up introducing new bugs and re-introducing
> old ones. a lot of complexity in well-used code comes from fixing things.
>
>> > Do you mean listEntry? listEntries is already taking a batch of UDSEntry
>> > objects so not much is won there. Modifying listEntry, however, to use the
>> > streaming method would be a win.
>>
>> No, i mean listEntries.
>> I also mean completely removing the SlaveBase::listEntry call from
>> slaves. The slaves are filling a stream which would then be send to
>> the client by calling (the new proposed) SlaveBase::listEntries.
>
> as long as SlaveBase::listEntry remains so as to not break 3rd party slaves
> (and ones you don't notice lurking in git.kde.org ;) that sounds great.
>
>> What i mean here is that the current listEntries function that takes a
>> UDSEntryList can be modified to use the new path.
>
> not sure that would win much; probably even be marginally slower due to more
> function calls. it would at least be using the exact same code path as a slave
> using the new API, which would be a consistency win ...
>
>> > (perhaps even a back-off algo -> first time-out is 50ms; if a buffer gets
>> > sent due to timeout then back-off to 100; repeat until you hit 300 ..
>> > easy to implement and should hopefully get first items to the client
>> > quicker)
>> This is theory and it works very well. In theory!
>> In practice things work a bit differently.
>>
>> The gui seems to be stuttering _while_ new entries flow in. Test it in
>
> yeah, this is an entirely different issue that while need an entirely different
> set of optimizations .. which probably won't be optimizations at all in terms
> of CPU usage but will result in more fluid UI.
>
>> dolphin on a massive folder. In fact, those that use kio::listdir for
>> listing folders only have interfaces that become usable when all
>> entries are fetched.
>
> assuming these UIs receive maintenance over the next years, this pattern is
> likely to fade away. instant feedback and incremental listing is a current
> reality.
>
> you know what would be very nice (but rather complex to achieve...)? for the
> client side of a KIO listing to say "i only want the first N items anyways,
> i'll let you know when i can handle more..." and for sorting to be optionally
> done on the slave side.

That would be very nice since you would cut IPC communication
massively (depending on the folder) to just those that you see which
would result in even insanely large folders to be presented to the
user near instantly.

But don't you just move logic to the slave that way? And lose
flexibility in the apps using the slave (like dolphin? Oh and
complicating kio "a bit" to pass a sorting and/or grouping style.
Right now it's not really difficult to add a completely new sort order
in dolphin, but that becomes very difficult when you want to let the
slave do the sorting.

Now imagine you actually have this feature implemented and dolphin (or
Accretion) adds a new sorting way that isn't in the slave Then you're
back to square one and perhaps even slower then you are right now.

Also, for a slave to give you the n items in a sorted way requires the
slave to fetch _all_ items to do the sorting. There is no way around
that. A trick like i'm doing in Accretion with nth_element +
partial_sort to sort just the items in my current view works because i
have all other items. All that you will save is IPC traffic. It might
not even be faster. Take a slow ftp for example. Without "slave side
sorting" you would get your first results after 300ms, guaranteed.
With server side sorting you will have to wait however long the slave
takes to fetch _all_ items. Then you can do a (partial) sort and send
it to the client.

I am not against this idea. Not at all, but i don't see how to
implement this in a way that it is always equal or faster then the
current situation.
>
> it is pretty ironic that a UI to show, say, videos from youtube will populate
> at least as smoothly, and on crappy enough hardware even better than, listing
> local directory contents because of this kind of incremental fetch and server-
> side pre-processing.

That is exactly the thing i'm taking on right now. I want everything
to be as fast as possible and present the user with data asap. In my
opinion there should be no reason (at all) to have desktop based
applications be slower then websites.
>
> KIO listing is all-or-nothing batch oriented; a stream-based approach that
> supports seeking through listings that are pre-sorted/grouped in the slave
> process would be moderately gorgeous. it would prevent more IPC than necessary
> and allow the slave to use any&all service-specific features for pre-sort/group
> of entries.

So you would save the stream in the slave side? But how would you
change the sorting if you just have a stream? Parse all data, sort it
and put it in a stream again?

>
>> Unless you can draft up a simple gui that work fast and fluid _while_
>> entries flow in.
>
> that's easy to do with QML. we have numerous examples of this in plasma
> active, in fact. the real trick is ensuring that your entries come to you pre-
> arranged so you don't show them moving around to the user.

If you have a view - where you don't scroll - then you're right. Then
it's possible.
If you have a view where entries flow in _while_ you are scrolling..
well.. then you're screwed. The current listview just stutters when
you do a trick like that (and yes, my pc is fast enough). I'm guessing
it's doing a lot of re calculations and repaints when entries flow in
and you move through them..?

>
>> > .. you'll also find some perhaps useful cleanups and optimizations in the
>> > aseigo/cleanups branch in kio. feel free to cherry-pick.
>>
>> You seem to like KIO :)
>
> well, i use it a lot :)
>
> --
> Aaron J. Seigo
> _______________________________________________
> Kde-frameworks-devel mailing list
> Kde-frameworks-devel at kde.org
> https://mail.kde.org/mailman/listinfo/kde-frameworks-devel
>