Review Request: New KIO::http_post and KIO::StoredHttpPost APIs that accept a QIODevice as input...

[Dawit A]

On Fri, Feb 4, 2011 at 11:34 AM, Thiago Macieira <thiago at kde.org> wrote:
> Em sexta-feira, 4 de fevereiro de 2011, às 16:53:14, David Faure escreveu:
>> QIODevice is a "pull" solution - you can ask for "1MB of data now".
>> Well, at least with buffers and files, not necessarily with sockets :-)
>
> QIODevice can work in that way too, since Qt 4.4.
>
> If you try to read and come up empty, you can just wait for there to be more
> data via QIODevice::readyRead().
>
>> The good thing with QIODevice is that it works on top of both memory
>> (QBuffer) and files (QFile), so in kde5 we could even remove the QByteArray
>> overload. And if you want to send bzip2 compressed data you could even
>> insert a KFilterDev to do it "on the fly".
>
> The problem of removing the QByteArray overload is that you don't know whether
> the QIODevice you've got has a fixed size or not. Upload operations often
> require an up-front byte count, so a missing size will require a full
> buffering.

Right, but why would that prevent us from removing the QByteArray
overload in the next kde 5.x as David suggested ?

We can simply enforce that the client, which most definitely would be
aware of the size of the data it wants to post, should tell use that
information by adding an additional size parameter to the API or
through a new meta-data "content-length". Also as you pointed out
below, the ioslave must guard against re-posting regardless of whether
or not the size information is available. Therefore, it should have no
problem determining the content size on its own if that information is
missing, which btw is exactly what kio_http does now even though the
current KIO job api could easily supply it the size information.
Instead kio_http stores the entire content to be sent in its own
QByteArray buffer, determines the size, and foist the entire content
onto the server in one single transmission!

> Another problem with QIODevice is re-posting. This can happen several times if
> your proxy requires authentication and your target requires authentication.
> QNetworkAccessManager will buffer by default, unless you tell it otherwise.

Right. And there are a couple of ways to address this issue, but both
of them have their pitfalls.

The first one is that the http ioslave can save the content to file if
it knows its size before hand and determines that its it too large to
hold in memory. The problems with doing this are the inefficiency of
reading from as well as writing to a file and the privacy/security
concerns of leaving the post content behind in case of a crash.

The other solution is simply to add an API that only accepts a
filename. The client apps that use this API would then have to write
the post data to some file and send the file name instead of the
content. This would solve the issue of too much memory use on both
sides of the equation, the job as well as the ioslave. However,
besides having the same downfall as the first solution this one is
also rather inefficient when posting small content, which is the
majority use case.

Regards,
Dawit A.