I take back what I said on sync(). It seems to be important for the program to recognize that the netCDF file is growing. On the netCDF writing side, a sync() command is equivalent to flushing an I/O buffer and this NC_SHARE setting tries to buffer flush optimally and automatically. I guess on the reading side it's more like an update of the file and record sizes.<div>
<br></div><div>I thought that I had tested this on a growing file, but apparently not. My other suggestion for the Handbook stands.<div><br></div><div>Yours,<br><div><br><div>Tom<br><br><div class="gmail_quote">On Tue, May 11, 2010 at 1:56 PM, Tom Downes <span dir="ltr"><<a href="mailto:tpdownes@caltech.edu">tpdownes@caltech.edu</a>></span> wrote:</div>
<div class="gmail_quote"><br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">Nicolas / Barth:<div><br></div><div>I made a number of modifications to the netCDF reader. The primary one is regarding the matrix problem I was having. Read the comment below for details.</div>
<div><br></div><div>The other potentially significant change I made is that I commented out a sync() command for the netCDF file. My understanding is that, from a performance perspective, it is much better for the user to write to the NC file while enabling the NC_SHARE mode, which attempts to optimize syncing. In fact, I think the sync command might be disabled if NC_SHARE is on. Anyhow, I would suggest adding NC_SHARE to KST's documentation on reading netCDF.</div>
<div><br></div><div>In a similar vein, I suggest editing the description of the update() datasource function on p. 203 of the KST handbook:</div><div><br></div><div>"this function should read any new data..."</div>
<div><br></div><div>Should instead read</div><div><br></div><div>"this function should see if new data exists and update the frame count and other member variables appropriately..."</div><div><div><br></div><div>
update() does not actually read data so it should not be suggested to the user that it does.</div><div><br></div><div>I'm not quite sure this code is ready for prime-time but if you have a chance to look at the changes, I would welcome comments. In particular, I think I may have broken the INDEX field somehow. I can't seem to create plots that have it as an x-axis vector, whereas everything else works just fine.</div>
<div><br></div><div><div> // TPD 10 May 2010 If the netCDF variable has 2 dimensions, then</div><div> // look to see if that variable has an attribute named "matrix"</div><div> // with a value of 1. If so, treat the variable as a set of vectors</div>
<div> // rather than as a set of frames with multiple samples per frame</div><div> // (the ability to do this was introduced by K. Scott in her 2005</div><div> // revision). This code shouldn't break the frame concept option.</div>
<div> //</div><div> // netCDF imposes the restriction that the UNLIMITED</div><div> // dimension (typically time) must come first in the dimension list</div><div> // so the second dimensions ends up being the number of rows.</div>
<div> //</div><div> // Subsequent functions (readField, update, etc) will recognize the</div><div> // fields as being from a matrix by the existence of a '/' character</div><div> // in the field name with the netCDF variable name ot the left and</div>
<div> // the row number .</div><div><br></div></div><font color="#888888"><div>Tom</div></font><div><div></div><div class="h5"><div><br><div class="gmail_quote">On Thu, Apr 22, 2010 at 8:40 AM, Tom Downes <span dir="ltr"><<a href="mailto:tpdownes@caltech.edu" target="_blank">tpdownes@caltech.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">I write the data as a matrix because I receive the data as UDP packets of 1xN_detectors every 10ms (100Hz sampling). It seems more efficient to do 1 write-to-disk of N_detectors samples than N_detectors writes of 1 sample - but I haven't actually tested this. The existing analysis software assumes a matrix so that is another reason to keep it that way.<br>
<br>I am working in Matlab which doesn't provide for user-accessible multi-threading and therefore makes building a set of FIFO buffers more difficult. Matlab so far has proven to be able to handle being a UDP client without losing data. I'd rather not have to build a whole UDP client in C or C++ where dividing the 1xN_detectors samples into N_detectors FIFO buffers would be easier.<br>
<br>The only metadata I care about are scaling factors and offsets for conversion to physical units, which are the same for all detectors. I don't think matrices are all that uncommon a way of doing things in NetCDF land - half the point of NetCDF is to be able to expand the dimensions of your matrices in a manner that is both efficient to store and to read.<br>
<br>KST 2b2 wouldn't even recognize a NetCDF file for what it was and I couldn't get any of the SVN checkouts to compile. So I installed KDE3 alongside KDE4 - works just fine. All you need are the kde3 libraries - you can continue to login using KDE4, if that's your preference.<br>
<br>I'll look at the datasource code and I'll be sure to share any work I do - since the Matlab UDP client is working brilliantly, it may turn out to be simpler to build a very stupid C program that just re-presents the data to KST.<br>
<br>Thanks for getting back,<br><font color="#888888"><br>Tom</font><div><div></div><div><br><br><div class="gmail_quote">On Thu, Apr 22, 2010 at 7:17 AM, Brisset, Nicolas <span dir="ltr"><<a href="mailto:Nicolas.Brisset@eurocopter.com" target="_blank">Nicolas.Brisset@eurocopter.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204, 204, 204);padding-left:1ex">
<div link="blue" vlink="purple" lang="DE">
<div>
<p class="MsoNormal"><font color="navy" face="Arial" size="2"><span style="font-size:10pt;font-family:Arial;color:navy">Hi Tom,</span></font></p>
<p class="MsoNormal"><font color="navy" face="Arial" size="2"><span style="font-size:10pt;font-family:Arial;color:navy"> </span></font></p>
<p class="MsoNormal"><font color="navy" face="Arial" size="2"><span style="font-size:10pt;font-family:Arial;color:navy" lang="EN-GB">the netcdf datasource for
kst currently does not support reading in matrices. It should however be fairly
easy to add if you feel so inclined, knowing that matrices are less well
supported than vectors as far as I know. I myself have only worked with vectors
up to now, so I may not be aware of everything. But I fear there is no easy way
to slice matrices into vectors (). Plus it makes little sense to me, as each variable
may have metadata and such attached to it and you would lose all that by
grouping vectors into a matrix. That said, I don't know your constraints well
enough. Maybe you have a good reason indeed to do it that way.</span></font></p>
<p class="MsoNormal"><font color="navy" face="Arial" size="2"><span style="font-size:10pt;font-family:Arial;color:navy" lang="EN-GB">Another path you may want
to go is making some changes to the netcdf datasource so that when it finds
matrices it provides their lines and columns as extra vectors. In that case all
the work should be done at datasource level, but that definitely sounds
feasible. You could query the dimensions of your 2-dimensional variables and
add to the field list vectors like Matrix1/Column1, Matrix1/Column2, Matrix1/Line1,
etc…</span></font></p>
<p class="MsoNormal"><font color="navy" face="Arial" size="2"><span style="font-size:10pt;font-family:Arial;color:navy" lang="EN-GB">Note: in the current code,
it seems that the dimensions of netcdf vars are not checked, that is *<b><span style="font-weight:bold">bad</span></b>*. For cdf, it is done better.</span></font></p>
<p class="MsoNormal"><font color="navy" face="Arial" size="2"><span style="font-size:10pt;font-family:Arial;color:navy" lang="EN-GB"> </span></font></p>
<p class="MsoNormal"><font color="navy" face="Arial" size="2"><span style="font-size:10pt;font-family:Arial;color:navy" lang="EN-GB">Then, the other question
you may ask yourself is whether to use kst 1.9.x or 2.x for that work. Kst 2.x
is still beta quality, but starting to work and there have been quite a few
changes (among others wrt datasources) so that what you do for one version of
the netcdf datasource may require substantial porting to the other one. FYI,
kst2 also works under Windows but nobody has yet tried netcdf under Windows. So
it is still definitely experimental.</span></font></p>
<p class="MsoNormal"><font color="navy" face="Arial" size="2"><span style="font-size:10pt;font-family:Arial;color:navy" lang="EN-GB"> </span></font></p>
<p class="MsoNormal"><font color="navy" face="Arial" size="2"><span style="font-size:10pt;font-family:Arial;color:navy" lang="EN-GB">Regards,</span></font></p>
<p class="MsoNormal"><font color="navy" face="Arial" size="2"><span style="font-size:10pt;font-family:Arial;color:navy" lang="EN-GB"> </span></font></p>
<p class="MsoNormal"><font color="navy" face="Arial" size="2"><span style="font-size:10pt;font-family:Arial;color:navy" lang="EN-GB">Nicolas</span></font></p>
<p class="MsoNormal"><font color="navy" face="Arial" size="2"><span style="font-size:10pt;font-family:Arial;color:navy" lang="EN-GB"> </span></font></p>
<div style="border-width:medium medium medium 1.5pt;border-style:none none none solid;border-color:-moz-use-text-color -moz-use-text-color -moz-use-text-color blue;padding:0cm 0cm 0cm 4pt">
<div>
<br><table style="border-collapse:collapse" border="0" width="38%">
<tbody><tr>
<td height="8" width="468"><b><font face="Arial" size="1">EUROCOPTER DEUTSCHLAND GmbH</font></b></td>
</tr>
<tr>
<td height="8" width="468"><b><font face="Arial" size="1">Sitz der Gesellschaft / Registered Office: Donauwörth</font></b></td>
</tr>
<tr>
<td height="8" width="468"><b><font face="Arial" size="1">Registergericht / Registration Court: Amtsgericht Augsburg HRB 16508</font></b></td>
</tr>
<tr>
<td height="8" width="468"><b><font face="Arial" size="1">Vorsitzender des Aufsichtsrates / Chairman of the Supervisory Board: Dr. Lutz Bertling</font></b></td>
</tr>
<tr>
<td height="8" width="468"><b><font face="Arial" size="1">Geschäftsführung / Board of Management:</font></b></td>
</tr>
<tr>
<td height="8" width="468"><b><font face="Arial" size="1">Dr. Wolfgang Schoder, Vorsitzender / CEO; Friedrich-Wilhelm Hormel; Ralf Barnscheidt</font></b></td>
</tr>
<tr>
<td style="border-bottom-style:double;border-bottom-width:3px" height="8" width="468"><font face="Arial" size="1"> </font></td>
</tr>
</tbody></table>
<table style="border-collapse:collapse" border="0" height="8" width="688">
<tbody><tr>
<td height="8" width="686"><font face="Arial" size="1"> </font></td>
</tr>
<tr>
<td height="8" width="686"><font face="Arial" size="1">CONFIDENTIALITY NOTICE</font></td>
</tr>
<tr>
<td height="8" width="686"><font face="Arial" size="1"> </font></td>
</tr>
<tr>
<td height="8" width="686"><font face="Arial" size="1">This communication and the information it contains is intended for the addressee ( s ) named above and for no other persons or organizations. </font></td>
</tr>
<tr>
<td height="8" width="686"><font face="Arial" size="1">It is confidential and may be legally privileged and protected by law. The unauthorized use, copying or disclosure of this communication </font></td>
</tr>
<tr>
<td height="8" width="686"><font face="Arial" size="1">or any part of it is prohibited and may be unlawful. </font></td>
</tr>
<tr>
<td height="8" width="686"><font face="Arial" size="1">If you have received this communication in error, kindly notify us by return e-mail and discard and/or delete the communication. </font></td>
</tr>
<tr>
<td height="8" width="686"><font face="Arial" size="1">Thank you very much.</font></td>
</tr>
<tr>
<td height="8" width="686"><font face="Arial" size="1">It is possible for e-mails to be intercepted or affected by viruses. Whilst we maintain virus checks on our e-mails, we accept no liability for viruses </font></td>
</tr>
<tr>
<td height="8" width="686"><font face="Arial" size="1">or other material which might be introduced with this message.</font></td>
</tr>
</tbody></table><br><div class="MsoNormal" style="text-align:center" align="center"><font face="Times New Roman" size="3"><span style="font-size:12pt">
<hr align="center" size="2" width="100%">
</span></font></div>
<p class="MsoNormal"><b><font face="Tahoma" size="2"><span style="font-size:10pt;font-family:Tahoma;font-weight:bold">Von:</span></font></b><font face="Tahoma" size="2"><span style="font-size:10pt;font-family:Tahoma"> Tom Downes
[mailto:<a href="mailto:tpdownes@caltech.edu" target="_blank">tpdownes@caltech.edu</a>] <br>
<b><span style="font-weight:bold">Gesendet:</span></b> Montag, 19. April 2010
20:46<br>
<b><span style="font-weight:bold">An:</span></b> <a href="mailto:kst@kde.org" target="_blank">kst@kde.org</a><br>
<b><span style="font-weight:bold">Betreff:</span></b> [Kst] reading in NetCDF
matrix (2-dim vector)</span></font></p>
</div><div><div></div><div>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size:12pt"> </span></font></p>
<div>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size:12pt">Hi:</span></font></p>
</div>
<div>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size:12pt"> </span></font></p>
</div>
<div>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size:12pt">I'm reasonably familiar with KST but am having a lot of trouble parsing
a NetCDF file in the fashion I intend. I am working on an instrument that
will have have around 100 independent detectors measured at 100 samples per
second each (i.e., around 10,000 total measurements per second). From the
perspective of data acquisition it is much easier to write to a NetCDF variable
with dimensions n_detectors x n_samples, where n_detectors would be this number
around 100 (126 in example below) and n_samples would equal 100 for 1 seconds
worth of data.</span></font></p>
</div>
<div>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size:12pt"> </span></font></p>
</div>
<div>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size:12pt">But trying to get KST to read this data in the way I intend seems to be
difficult. I want it to be able to split the matrix into 126 separate
vectors so that I can plot a timestream for each - realistically I would take a
representative sample. The "sample" and "frame"
terminology doesn't seem to quite match up to what I am doing - frames seem to
be more about bookkeeping than actually combining many timestream vectors into
a matrix.</span></font></p>
</div>
<div>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size:12pt"> </span></font></p>
</div>
<div>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size:12pt">Anyhow, I would like to solve this in a relatively simple way, with a
preference toward avoiding re-writing how the data is originally written to
disk as it is much simpler this way. Can I do what I need to do within
KST or with a simple plug-in? I am quite comfortable working in
C/C++/Java and shell scripting environments. Suggestions?</span></font></p>
</div>
<div>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size:12pt"> </span></font></p>
</div>
<div>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size:12pt">This is what the file looks like:</span></font></p>
</div>
<div>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size:12pt"> </span></font></p>
</div>
<div>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size:12pt">netcdf test-udp {</span></font></p>
</div>
<div>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size:12pt">dimensions:</span></font></p>
</div>
<div>
<p class="MsoNormal"><span><font face="Times New Roman" size="3"><span style="font-size:12pt"> </span></font></span>time
= UNLIMITED ; // (3960037 currently)</p>
</div>
<div>
<p class="MsoNormal"><span><font face="Times New Roman" size="3"><span style="font-size:12pt"> </span></font></span>nkids
= 126 ;</p>
</div>
<div>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size:12pt">variables:</span></font></p>
</div>
<div>
<p class="MsoNormal"><span><font face="Times New Roman" size="3"><span style="font-size:12pt"> </span></font></span>int
kids(time, nkids) ;</p>
</div>
<div>
<p class="MsoNormal"><span><font face="Times New Roman" size="3"><span style="font-size:12pt"> </span></font></span>kids:scale_factor
= 1 ;</p>
</div>
<div>
<p class="MsoNormal"><span><font face="Times New Roman" size="3"><span style="font-size:12pt"> </span></font></span>kids:add_offset
= 0 ;</p>
</div>
<div>
<p class="MsoNormal"><span><font face="Times New Roman" size="3"><span style="font-size:12pt"> </span></font></span>kids:units
= "bits" ;</p>
</div>
<div>
<p class="MsoNormal"><span><font face="Times New Roman" size="3"><span style="font-size:12pt"> </span></font></span>int
timestamp(time) ;</p>
</div>
<div>
<p class="MsoNormal"><span><font face="Times New Roman" size="3"><span style="font-size:12pt"> </span></font></span>timestamp:scale_factor
= 1 ;</p>
</div>
<div>
<p class="MsoNormal"><span><font face="Times New Roman" size="3"><span style="font-size:12pt"> </span></font></span>timestamp:add_offset
= 0 ;</p>
</div>
<div>
<p class="MsoNormal"><span><font face="Times New Roman" size="3"><span style="font-size:12pt"> </span></font></span>timestamp:units
= "bits" ;</p>
</div>
<div>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size:12pt">}</span></font></p>
</div>
<div>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size:12pt"> </span></font></p>
</div>
<div>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size:12pt">I am running KST 1.9.1 compiled manually after checking out the release
source. I don't think it matters, but I am running the kde3 libraries
alongside the kde4 libraries while logged into a KDE 3.5 WM session in OpenSUSE
11.2.</span></font></p>
</div>
<div>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size:12pt"> </span></font></p>
</div>
<div>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size:12pt">Tom</span></font></p>
</div>
</div></div></div>
</div>
</div>
</blockquote></div><br>
</div></div></blockquote></div><br></div></div></div></div>
</blockquote></div><br></div></div></div></div>