KIO directory listing - CPU slows down SSD

[Aaron J. Seigo]

On Thursday, May 29, 2014 16:32:28 Mark Gaiser wrote:
> On Thu, May 29, 2014 at 12:21 AM, Aaron J. Seigo <ase
 
> It's just a hobby for me to optimize it to levels that match raw speed
> (without any KIO).

meaningful speed improvements would be a reason to change things.

if wire compatibility can be maintained while reaping speed benefits (and i 
think we've seen it can be done ...), then that isn't a reason.

> I'm "trying" to improve things by making them simpler. If that
> requires a new code path: fine with me.

simplicity is a good goal, too. personally, i'd count it as a minor goal when 
looking at a long-maintained/debugged codebase, though. disturbing too much in 
the quest for simplicity often ends up introducing new bugs and re-introducing 
old ones. a lot of complexity in well-used code comes from fixing things.

> > Do you mean listEntry? listEntries is already taking a batch of UDSEntry
> > objects so not much is won there. Modifying listEntry, however, to use the
> > streaming method would be a win.
> 
> No, i mean listEntries.
> I also mean completely removing the SlaveBase::listEntry call from
> slaves. The slaves are filling a stream which would then be send to
> the client by calling (the new proposed) SlaveBase::listEntries.

as long as SlaveBase::listEntry remains so as to not break 3rd party slaves 
(and ones you don't notice lurking in git.kde.org ;) that sounds great.

> What i mean here is that the current listEntries function that takes a
> UDSEntryList can be modified to use the new path.

not sure that would win much; probably even be marginally slower due to more 
function calls. it would at least be using the exact same code path as a slave 
using the new API, which would be a consistency win ...

> > (perhaps even a back-off algo -> first time-out is 50ms; if a buffer gets
> > sent due to timeout then back-off to 100; repeat until you hit 300 ..
> > easy to implement and should hopefully get first items to the client
> > quicker)
> This is theory and it works very well. In theory!
> In practice things work a bit differently.
> 
> The gui seems to be stuttering _while_ new entries flow in. Test it in

yeah, this is an entirely different issue that while need an entirely different 
set of optimizations .. which probably won't be optimizations at all in terms 
of CPU usage but will result in more fluid UI.

> dolphin on a massive folder. In fact, those that use kio::listdir for
> listing folders only have interfaces that become usable when all
> entries are fetched. 

assuming these UIs receive maintenance over the next years, this pattern is 
likely to fade away. instant feedback and incremental listing is a current 
reality.

you know what would be very nice (but rather complex to achieve...)? for the 
client side of a KIO listing to say "i only want the first N items anyways, 
i'll let you know when i can handle more..." and for sorting to be optionally 
done on the slave side.

it is pretty ironic that a UI to show, say, videos from youtube will populate 
at least as smoothly, and on crappy enough hardware even better than, listing 
local directory contents because of this kind of incremental fetch and server-
side pre-processing.

KIO listing is all-or-nothing batch oriented; a stream-based approach that 
supports seeking through listings that are pre-sorted/grouped in the slave 
process would be moderately gorgeous. it would prevent more IPC than necessary 
and allow the slave to use any&all service-specific features for pre-sort/group 
of entries.
 
> Unless you can draft up a simple gui that work fast and fluid _while_
> entries flow in.

that's easy to do with QML. we have numerous examples of this in plasma 
active, in fact. the real trick is ensuring that your entries come to you pre-
arranged so you don't show them moving around to the user.

> > .. you'll also find some perhaps useful cleanups and optimizations in the
> > aseigo/cleanups branch in kio. feel free to cherry-pick.
> 
> You seem to like KIO :)

well, i use it a lot :)

-- 
Aaron J. Seigo
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.kde.org/pipermail/kde-frameworks-devel/attachments/20140602/0729d503/attachment.sig>