KIO directory listing - CPU slows down SSD

[Mark Gaiser]

On Mon, Jun 2, 2014 at 11:39 PM, Frank Reininghaus
<frank78ac at googlemail.com> wrote:
> Hi,
>
> 2014-06-02 20:54 GMT+02:00 Mark Gaiser:
>> On Mon, Jun 2, 2014 at 6:42 PM, Aaron J. Seigo wrote:
>>> On Thursday, May 29, 2014 16:32:28 Mark Gaiser wrote:
>>>> dolphin on a massive folder. In fact, those that use kio::listdir for
>>>> listing folders only have interfaces that become usable when all
>>>> entries are fetched.
>>>
>>> assuming these UIs receive maintenance over the next years, this pattern is
>>> likely to fade away. instant feedback and incremental listing is a current
>>> reality.
>>>
>>> you know what would be very nice (but rather complex to achieve...)? for the
>>> client side of a KIO listing to say "i only want the first N items anyways,
>>> i'll let you know when i can handle more..." and for sorting to be optionally
>>> done on the slave side.
>>
>> That would be very nice since you would cut IPC communication
>> massively (depending on the folder) to just those that you see which
>> would result in even insanely large folders to be presented to the
>> user near instantly.
>>
>> But don't you just move logic to the slave that way? And lose
>> flexibility in the apps using the slave (like dolphin? Oh and
>> complicating kio "a bit" to pass a sorting and/or grouping style.
>> Right now it's not really difficult to add a completely new sort order
>> in dolphin, but that becomes very difficult when you want to let the
>> slave do the sorting.
>
> Yes, such an approach would indeed require additional complexity in
> KIO and the slaves and would sacrifice flexibility.
>
> Therefore, it might be questionable if this is really the way to go -
> I didn't really mean that this should be implemented when I said that
> I had also thought about this approach in my reply to Aaron.
> Sometimes, I just dream and think about things that we could do if we
> had infinite resources ;-)

:) same here.

We should do this as a gsoc project. Since this would be very
interesting! And while typing that i think: "I want to be a student
again. Just to do that gsoc."
>
> I've seen users complain that loading huge folders is much faster in,
> e.g., Windows Explorer than in Dolphin (that was some time ago though,
> we might have caught up a bit in the mean time). Compared to us,
> Microsoft's and Apple's resources are, to a good approximation,
> infinite, so I always imagined that they would use every conceivable
> trick to make their file managers faster.

I don't know for mac, but i do for windows (kind of). I tried loading
a folder with 500.000 entries in windows. I don't remember the exact
measurements since i didn't wrote them down. I just did a visual
comparison. From that i vaguely remember:
- Windows explorer being a lot slower in loading and sorting 500.000
files then dolphin is (was with kde 4.13.0)
- Windows did use less memory then dolphin. No number, i just remember
it being less.

>
>> Now imagine you actually have this feature implemented and dolphin (or
>> Accretion) adds a new sorting way that isn't in the slave Then you're
>> back to square one and perhaps even slower then you are right now.
>>
>> Also, for a slave to give you the n items in a sorted way requires the
>> slave to fetch _all_ items to do the sorting.
>
> Yes, but the slave will only have to fetch the file names for all
> items (provided that we sort by name), nothing else.

In this case it doesn't really matter what the slave wants. Dolphin
wants something as well and that "something" is at least a filename,
date and size. Thus it requires the slave to fetch all details
(details set at 2). The slave could do it in two passes though.
Pass 1: fetch the files with details set at 0 just for the sorting in
case it's set at filename
Pass 2: fetch the files with details set at 2 and send those over the
socket that are currently visible in the view.
>
> Cheers,
> Frank

/me keeps dreaming for a while since i like these ideas :)