Which KIO job is suitable for batching?

[Mark Gaiser]

Hi,

By batching i mean send a bunch of data to the IO slave.
I hoped KIO::BatchRenameJob would be a start for me as the name does imply
batching, but doesn't seem to be doing just that. It's merely that you call
it with a list of url's which then sequentially get processed inside
KIO::BatchRenameJob. What happens there (as far as i can tell from the
code) is after each successful rename a new rename gets fired till all the
url's have been handled.

KIO::SimpleJob is for just one url, that doesn't seem to be an option for
my usecase*.

SimpleJobPrivate is where the KIO magic really happens, but it takes only
one url (by design).
It does a lot of the heavy lifting and you'd probably not want to create a
job on any lower level then that. The "m_slave" (KIO::Slave) member
in SimpleJobPrivate the one i want to have access to, specifically for the
send(...) command in m_slave. I want to send a CMD_MULTI_GET command and
implement support for that in the thumbnail slave. But i don't quite see
how to get there with the current KIO::SimpleJob or KIO::Job classes.

I can see a few options, but i don't know how realistic those would be.

1. Split SimpleJobPrivate into a:
  - SimpleDataJobPrivate : public JobPrivate
  - SimpleJobPrivate : public SimpleDataJobPrivate
SimpleDataJobPrivate would get no knowledge of the QUrl logic that the
current SimpleJobPrivate has. It would contain all the connection and
scheduler handling.
SimpleJobPrivate would be stripped down to inherit from
SimpleDataJobPrivate and do what it does now. A public facing
"KIO::SimpleDataJob" would be exposed to the public API side.
The downside is that the current JobPrivate class has quite a bit of file
handling specifics that would be out of place for SimpleDataJobPrivate.
2. Expose a public send function in KIO::Job to send raw data to the
underlying slave with a given command. KIO::SimpleJob can then make that
function private, but making a new class that inherits from KIO::Job and
sending some data then becomes much easier. Though i'm not quite sure how
possible this is as i don't see where KIO::Job would get a KIO::Slave
object...
3. Leave the jobs as-is and implement a queue mechanism in the thumbnail
slave. So any subsequent calls to that slave while it's still processing an
image will end up on a queue. It's possible and the least invasive way i
can think of, but complicates the thumbnail slave side (possibly even
SlaveBase) and is probably not as performant as sending a whole bunch of
url's at once.

* = I'm trying to generate thumbnails as fast as possible. IO is not the
bottleneck. I can thumbnail 4000 4k images in 120 seconds in sequential
order which is reduced to ~22 seconds in parallel. It's not a perfect
optimization but much faster. But for this to work in KIO the IOSlave needs
to receive a bunch of urls to process.

Cheers,
Mark
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.kde.org/pipermail/kde-core-devel/attachments/20180902/4891147f/attachment.htm>