[Kst] branches/work/kst/portto4/kst/src/datasources/ascii

[Peter Kümmel]

On 17.10.2012 00:19, Brisset, Nicolas wrote:
> OK, thanks for the explanations.
> So instead of disabling it when we read from one file, maybe we should disable it when the whole file does not fit into the allowed memory?
> A crash is always bad, so we have to be careful there. Is there a way to predict or catch the crash and restart without threading?
> Otherwise we may want to disable multithreading on a single file altogether.

The last patch is good enough: we disable threads for sliding windows.
And the sliding window must be enabled manually.

When reading the whole file it should succeed or fail before we start any thread.
(this was the idea, but should be tested again)

>
> I hope Barth will answer the UI question (where to put the ASCII reading options) so that we can finalize this first step and publish a 2.0.7 to test extensively.
>
> Nicolas
>
>> -----Ursprüngliche Nachricht-----
>> Von: kst-bounces at kde.org [mailto:kst-bounces at kde.org] Im Auftrag von Peter
>> Kümmel
>> Gesendet: Mittwoch, 17. Oktober 2012 00:04
>> An: kst at kde.org
>> Betreff: Re: [Kst] branches/work/kst/portto4/kst/src/datasources/ascii
>>
>> On 16.10.2012 23:16, Brisset, Nicolas wrote:
>>>> SVN commit 1320822 by kuemmel:
>>>>
>>>> don't read with mutiple threads from the same file
>>> Does that mean the speedup you measured applies only to the case where we
>> read from many files simultaneously?
>>>   From the figures you posted it sounded like you had gotten it to run on a
>> single file...
>>
>> Here it was also one file, but it must be read as whole file, not with a sliding
>> window.
>>
>> The ascii plugin reads the file first into memory, then it parses for the data.
>> Without using a sliding window we have the whole file splitted into several
>> parts in memory which we could parse in parallel.
>>
>> But when we use a sliding window, the data is only read from disk when we
>> wanna access the data while parsing. And when we start multiple threads to
>> parse these parts we effectively try to read from the same file with several
>> threads, therefore the crash.
>>
>>
>> When you wanna see the speedup you have to use a file which fits into memory
>> (don't enable the limitation of the file buffer).
>>
>> It's possible to also use multiple threads when using a sliding window, but then
>> we have to split the window and have to start threads on these window parts.
>> I will add this within the next days.
>>
>> Peter
>>
>>
>>
>> _______________________________________________
>> Kst mailing list
>> Kst at kde.org
>> https://mail.kde.org/mailman/listinfo/kst
> EUROCOPTER DEUTSCHLAND GmbH
> Sitz der Gesellschaft / Registered Office: Donauwörth
> Registergericht / Registration Court: Amtsgericht Augsburg HRB 16508
> Vorsitzender des Aufsichtsrates / Chairman of the Supervisory Board: Dr. Lutz Bertling
> Geschäftsführung / Board of Management:
> Dr. Wolfgang Schoder, Vorsitzender / CEO; Friedrich-Wilhelm Hormel; Ralf Barnscheidt
>
>
> CONFIDENTIALITY NOTICE
>
> This communication and the information it contains is intended for the addressee ( s ) named above and for no other persons or organizations.
> It is confidential and may be legally privileged and protected by law. The unauthorized use, copying or disclosure of this communication
> or any part of it is prohibited and may be unlawful.
> If you have received this communication in error, kindly notify us by return e-mail and discard and/or delete the communication.
> Thank you very much.
> It is possible for e-mails to be intercepted or affected by viruses. Whilst we maintain virus checks on our e-mails, we accept no liability for viruses
> or other material which might be introduced with this message.
> _______________________________________________
> Kst mailing list
> Kst at kde.org
> https://mail.kde.org/mailman/listinfo/kst
>