<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<div class="moz-cite-prefix">Hi Peter,<br>
<br>
I've done some more testing with the latest 64 bit build and the
big ascii file. I have some questions and comments for you.<br>
<br>
Why does kst consume so much memory when loading this file? The
file contains 3 vectors, each vector requires around 3.2GB of
memory, a total of 9.6GB. But when I load this file with no buffer
limit kst uses 22GB of my 24GB available. I've tried various
buffer limits from 12GB down to 3MB and the best case memory usage
is 15GB.<br>
<br>
The buffer limit can have a dramatic influence on load times. In
general I've found the smaller the buffer the quicker it loads.
However, too small and kst will crash (loading big ascii file, kst
crashes with a buffer limit of 2MB and less). The best case load
time was 5min 18sec with a buffer limit of 5MB, the worst case was
20min 24sec with a buffer limit of 12GB. With the buffer limit
disabled the load time was 12min 36sec. Using a small memory
buffer is faster (up to 3 times) and uses less memory (about 30%).
Given that there is a lot to be gained by using a small memory
buffer are there any disadvantages I should be aware of? With such
a dramatic difference it would be great if kst could pick a
suitable buffer limit automatically.<br>
<br>
<img src="cid:part1.05090100.06010109@gmail.com" alt=""><br>
<br>
I've attached plots of memory usage for each of the buffer sizes I
tried. It is interesting to see how different they are above
500MB, given that the same file is used in all cases. It can
clearly be seen that there is a lot of inefficiency with a large
buffer.<br>
<br>
There seems to be a problem with the loading of the last (third)
column. It takes much longer to load than the previous two
columns. For example, with a 3MB buffer size the first two columns
load in about 40sec each while the last column takes about 200 sec
(5 times longer). Do you think there many be a problem with how
the last column is processed or is this to be expected?<br>
<br>
I really like the new status updates during the loading process,
however, I have one small suggestion. During the loading process,
after each column is read in the status bar indicates that each
column is being plotted before moving on to the next column,
however, nothing is rendered to the screen until after the last
column is processed. There may be a better way to describe the
"plotting data..." step because it looks like kst is not doing
what it says it's doing.<br>
<br>
When I plotted the above graph the new progress bar gets stuck at
50%. You can try it yourself with the attached data file "mem
data.csv".<br>
<br>
Regards, Ben<br>
<br>
<br>
On 7/12/2013 8:41 AM, Peter Kümmel wrote:<br>
</div>
<blockquote cite="mid:52A2446F.7050709@gmx.net" type="cite">On
06.12.2013 12:39, Ben Lewis wrote:
<br>
<blockquote type="cite">Hi Peter,
<br>
<br>
I can now open the big ASCII file using the settings you
recommended. :-)
<br>
<br>
Build: x64
<br>
Limit buffer size: 500MB
<br>
Use threads: Yes
<br>
Interpret empty value as: NULL
<br>
<br>
I have not tried other settings yet.
<br>
<br>
I can load all three columns in just under 10 minutes.
<br>
</blockquote>
<br>
When you have enough memory disable the buffer limit, then the
file is only read once.
<br>
With buffer limit enabled, for each column the file is read again,
and reading is
<br>
the bottleneck when you don't have a SSD.
<br>
<br>
<blockquote type="cite">
<br>
My only criticism is that the progress bar does not behave as
expected when loading multiple columns. When loading all
<br>
three columns I observe the following behaviour:
<br>
</blockquote>
<br>
Should be fixed now.
<br>
<br>
Cheers,
<br>
Peter
<br>
<br>
<blockquote type="cite">
<br>
Searching for rows: 0-50%
<br>
Reading data.../Parsing data.. 50-100% (quick)
<br>
Reading column 2: 50%
<br>
Reading data.../Parsing data... 50-100% (slow)
<br>
<br>
Once loaded, performance is a little slow with the full data set
displayed, but after zooming in performance is
<br>
excellent with smooth scrolling and zooming.
<br>
<br>
Regards, Ben
<br>
</blockquote>
<br>
<br>
_______________________________________________
<br>
Kst mailing list
<br>
<a class="moz-txt-link-abbreviated" href="mailto:Kst@kde.org">Kst@kde.org</a>
<br>
<a class="moz-txt-link-freetext" href="https://mail.kde.org/mailman/listinfo/kst">https://mail.kde.org/mailman/listinfo/kst</a>
<br>
</blockquote>
<br>
</body>
</html>