Failure while executing KTar::open while using KCompressionDevice as the device

[Luiz Romário Santana Rios]

Hello again.

2015-11-07 6:03 GMT-03:00 David Faure <faure at kde.org>:
> On Friday 06 November 2015 08:47:34 Luiz Romário Santana Rios wrote:
>> 2015-11-06 4:48 GMT-03:00 David Faure <faure at kde.org>:
>> > On Monday 02 November 2015 14:53:40 Luiz Romário Santana Rios wrote:
>> >>
>> >> I'm trying to decompress a XZ archive downloaded using
>> >> QNetworkAccessManager, so, according to the documents, I have to pass
>> >> the QNetworkReply pointer to a KCompressionDevice
>> >
>> > I don't think this can work at all.
>> > (and yes I've seen your review request, but while it fixes the file:/// case, are you sure
>> > it fixes the network case as well?)
>>
>> Since I wait for the QNAM::finished() signal before doing anything
>> with the QNetworkReply, it probably does, but maybe not for, say, a
>> QTcpSocket. That case would need the waitFor*() calls, indeed.
>
> Ah, I see.
>
>> > So the reason it breaks (apart from the issue of seeking) is that when KTar
>> > (or KCompressionDevice) wants to read more data, it might not be available,
>> > and the read fails. You could add waitFor* calls, but that would make the whole
>> > thing blocking - very bad for the main thread of a GUI program.
>>
>> It already blocks, even in the KTar archive("file.tar.gz") case.
>
> Blocking while uncompressing is OK (fast CPU operation),
More or less. In my machine, for instance, KTar::open() blocks for
about 9-10 secs when uncompressing a ~40 MiB .xz package compressed
with -9. But, yeah, definitely faster than waiting for a CPU packet.

> blocking while waiting for network packets is not (could take 30 minutes).
I think there are two options to handle this: warn the user that
KCompressionDevice assumes all data is available and return false from
KTar::open() if it's not, or make the waitFor* calls and warn the user
that passing a QIODevice which is not yet fully ready to
KCompressionDevice might make KTar::open() block.

> But now I see, you do an async download first, so no issue there.

>
>> Then we could use a temp file in the KZip implementation and this
>> approach for KTar?
>
> I wouldn't want this for the case of random access devices, like
> files on disk (it would be pointless to copy from a local file to a tempfile).
> But yes I'm not opposed to code that would use a temp file for
> cases of non-sequential devices being used as input.
>
> Aleix suggested QBuffer, but archives can be *huge*, so this might
> eat all available memory. This is the reason why KTar uses a temp file
> for the compressed-tar case.

A suggestion I made in the IRC channel was this: add a constructor
argument -- or a flag -- called "Buffered" or "Cached"; if enabled,
the data stream from the QIODevice would be buffered or copied to a
tempfile, then that buffer or file would be extracted; if disabled,
the QIODevice would be accessed directly for its data; then, make it
enabled by default.

Another thing: KCompressionDevice only seems to support tar files
(.gz, .bz2, and .xz), so why would KZip be an issue anyway?

>
> --
> David Faure, faure at kde.org, http://www.davidfaure.fr
> Working on KDE Frameworks 5
>



-- 
Luiz Romário Santana Rios