KIO directory listing - CPU slows down SSD

[Mark Gaiser]

On Wed, May 28, 2014 at 9:59 AM, David Faure <faure at kde.org> wrote:
> On Wednesday 14 May 2014 12:50:18 Aaron J. Seigo wrote:
>> * one entry at a time, serialized
>> * no changing data once put there
>> * no getting the data back out
>>
>> the class would then just batch up data into its QByteArray and then shunt
>> it  across the wire when needed. no middle men, no 100s of 1000s of objects
>> and QLists and << operators. just an API sth like:
>>
>>         startEntry
>>         addField
>>         endEntry
>
> I like this idea very much. Mark: see, it's quite close to what we were
> discussing by email recently. And I hadn't read this yet :)
>
> Note how this is not specific to the ioslave using QT_STATBUFF :)

To be honest, i was trying to look at the most used case (which is
kio_file) and optimize that for raw speed.
But i kinda like the idea of this approach as well since it would work
on all slaves :) I will try to implement that instead.

What i really like about this approach is that it could eventually
phase out UDSEntry from the slave side. Meaning that UDSEntry would be
fully client side (where this stream would be parsed).

>
>> internally it could either batch up the fields and then stream them out
>> when  endEntry is called, or (and this is what i would do personally) on
>> startEntry it would set up the header but with bad data (e.g. field count
>> of 0) and then start appending the fields as they come in and increment a
>> field count variable. when endEntry is called, update the header with the
>> right field count.
>
> Ah but batching requires a vector again. Or you meant writing into a bytearray
> and coming back to update the count - that would rely on QDataStream internals
> I'm afraid (how to know the offset....).
>
> But the alternative is to replace the field count with an "end of fields"
> special field.
>
Huh? "batching requires a vector again" .. i don't understand that. It
should work somewhat like this, right (heavily simplified, obviously):
SomeDataStreamClass stream;
int i = 0
while(QT_READDIR(...) && i++) {
  if(i = 200) { // or whatever the threshold is to send a batch of entries
     listEntries(stream);
     // reset stream and i.
  }
  startEntry()
  addField(...)
  addField(...)
  ...
  endEntry()
}

No need for a vector or any special things.
What is needed is a new listEntries(...) function that either takes a
QDataStream as argument or a QByteArray.

> --
> David Faure, faure at kde.org, http://www.davidfaure.fr
> Working on KDE Frameworks 5
>