I write the data as a matrix because I receive the data as UDP packets of 1xN_detectors every 10ms (100Hz sampling). It seems more efficient to do 1 write-to-disk of N_detectors samples than N_detectors writes of 1 sample - but I haven't actually tested this. The existing analysis software assumes a matrix so that is another reason to keep it that way.<br>
<br>I am working in Matlab which doesn't provide for user-accessible multi-threading and therefore makes building a set of FIFO buffers more difficult. Matlab so far has proven to be able to handle being a UDP client without losing data. I'd rather not have to build a whole UDP client in C or C++ where dividing the 1xN_detectors samples into N_detectors FIFO buffers would be easier.<br>
<br>The only metadata I care about are scaling factors and offsets for conversion to physical units, which are the same for all detectors. I don't think matrices are all that uncommon a way of doing things in NetCDF land - half the point of NetCDF is to be able to expand the dimensions of your matrices in a manner that is both efficient to store and to read.<br>
<br>KST 2b2 wouldn't even recognize a NetCDF file for what it was and I couldn't get any of the SVN checkouts to compile. So I installed KDE3 alongside KDE4 - works just fine. All you need are the kde3 libraries - you can continue to login using KDE4, if that's your preference.<br>
<br>I'll look at the datasource code and I'll be sure to share any work I do - since the Matlab UDP client is working brilliantly, it may turn out to be simpler to build a very stupid C program that just re-presents the data to KST.<br>
<br>Thanks for getting back,<br><br>Tom<br><br><div class="gmail_quote">On Thu, Apr 22, 2010 at 7:17 AM, Brisset, Nicolas <span dir="ltr"><<a href="mailto:Nicolas.Brisset@eurocopter.com">Nicolas.Brisset@eurocopter.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<div link="blue" vlink="purple" lang="DE">
<div>
<p class="MsoNormal"><font color="navy" face="Arial" size="2"><span style="font-size: 10pt; font-family: Arial; color: navy;">Hi Tom,</span></font></p>
<p class="MsoNormal"><font color="navy" face="Arial" size="2"><span style="font-size: 10pt; font-family: Arial; color: navy;"> </span></font></p>
<p class="MsoNormal"><font color="navy" face="Arial" size="2"><span style="font-size: 10pt; font-family: Arial; color: navy;" lang="EN-GB">the netcdf datasource for
kst currently does not support reading in matrices. It should however be fairly
easy to add if you feel so inclined, knowing that matrices are less well
supported than vectors as far as I know. I myself have only worked with vectors
up to now, so I may not be aware of everything. But I fear there is no easy way
to slice matrices into vectors (). Plus it makes little sense to me, as each variable
may have metadata and such attached to it and you would lose all that by
grouping vectors into a matrix. That said, I don't know your constraints well
enough. Maybe you have a good reason indeed to do it that way.</span></font></p>
<p class="MsoNormal"><font color="navy" face="Arial" size="2"><span style="font-size: 10pt; font-family: Arial; color: navy;" lang="EN-GB">Another path you may want
to go is making some changes to the netcdf datasource so that when it finds
matrices it provides their lines and columns as extra vectors. In that case all
the work should be done at datasource level, but that definitely sounds
feasible. You could query the dimensions of your 2-dimensional variables and
add to the field list vectors like Matrix1/Column1, Matrix1/Column2, Matrix1/Line1,
etc…</span></font></p>
<p class="MsoNormal"><font color="navy" face="Arial" size="2"><span style="font-size: 10pt; font-family: Arial; color: navy;" lang="EN-GB">Note: in the current code,
it seems that the dimensions of netcdf vars are not checked, that is *<b><span style="font-weight: bold;">bad</span></b>*. For cdf, it is done better.</span></font></p>
<p class="MsoNormal"><font color="navy" face="Arial" size="2"><span style="font-size: 10pt; font-family: Arial; color: navy;" lang="EN-GB"> </span></font></p>
<p class="MsoNormal"><font color="navy" face="Arial" size="2"><span style="font-size: 10pt; font-family: Arial; color: navy;" lang="EN-GB">Then, the other question
you may ask yourself is whether to use kst 1.9.x or 2.x for that work. Kst 2.x
is still beta quality, but starting to work and there have been quite a few
changes (among others wrt datasources) so that what you do for one version of
the netcdf datasource may require substantial porting to the other one. FYI,
kst2 also works under Windows but nobody has yet tried netcdf under Windows. So
it is still definitely experimental.</span></font></p>
<p class="MsoNormal"><font color="navy" face="Arial" size="2"><span style="font-size: 10pt; font-family: Arial; color: navy;" lang="EN-GB"> </span></font></p>
<p class="MsoNormal"><font color="navy" face="Arial" size="2"><span style="font-size: 10pt; font-family: Arial; color: navy;" lang="EN-GB">Regards,</span></font></p>
<p class="MsoNormal"><font color="navy" face="Arial" size="2"><span style="font-size: 10pt; font-family: Arial; color: navy;" lang="EN-GB"> </span></font></p>
<p class="MsoNormal"><font color="navy" face="Arial" size="2"><span style="font-size: 10pt; font-family: Arial; color: navy;" lang="EN-GB">Nicolas</span></font></p>
<p class="MsoNormal"><font color="navy" face="Arial" size="2"><span style="font-size: 10pt; font-family: Arial; color: navy;" lang="EN-GB"> </span></font></p>
<div style="border-width: medium medium medium 1.5pt; border-style: none none none solid; border-color: -moz-use-text-color -moz-use-text-color -moz-use-text-color blue; padding: 0cm 0cm 0cm 4pt;">
<div>
<br><table style="border-collapse: collapse;" border="0" width="38%">
<tbody><tr>
<td height="8" width="468"><b><font face="Arial" size="1">EUROCOPTER DEUTSCHLAND GmbH</font></b></td>
</tr>
<tr>
<td height="8" width="468"><b><font face="Arial" size="1">Sitz der Gesellschaft / Registered Office: Donauwörth</font></b></td>
</tr>
<tr>
<td height="8" width="468"><b><font face="Arial" size="1">Registergericht / Registration Court: Amtsgericht Augsburg HRB 16508</font></b></td>
</tr>
<tr>
<td height="8" width="468"><b><font face="Arial" size="1">Vorsitzender des Aufsichtsrates / Chairman of the Supervisory Board: Dr. Lutz Bertling</font></b></td>
</tr>
<tr>
<td height="8" width="468"><b><font face="Arial" size="1">Geschäftsführung / Board of Management:</font></b></td>
</tr>
<tr>
<td height="8" width="468"><b><font face="Arial" size="1">Dr. Wolfgang Schoder, Vorsitzender / CEO; Friedrich-Wilhelm Hormel; Ralf Barnscheidt</font></b></td>
</tr>
<tr>
<td style="border-bottom-style: double; border-bottom-width: 3px;" height="8" width="468"><font face="Arial" size="1"> </font></td>
</tr>
</tbody></table>
<table style="border-collapse: collapse;" border="0" height="8" width="688">
<tbody><tr>
<td height="8" width="686"><font face="Arial" size="1"> </font></td>
</tr>
<tr>
<td height="8" width="686"><font face="Arial" size="1">CONFIDENTIALITY NOTICE</font></td>
</tr>
<tr>
<td height="8" width="686"><font face="Arial" size="1"> </font></td>
</tr>
<tr>
<td height="8" width="686"><font face="Arial" size="1">This communication and the information it contains is intended for the addressee ( s ) named above and for no other persons or organizations. </font></td>
</tr>
<tr>
<td height="8" width="686"><font face="Arial" size="1">It is confidential and may be legally privileged and protected by law. The unauthorized use, copying or disclosure of this communication </font></td>
</tr>
<tr>
<td height="8" width="686"><font face="Arial" size="1">or any part of it is prohibited and may be unlawful. </font></td>
</tr>
<tr>
<td height="8" width="686"><font face="Arial" size="1">If you have received this communication in error, kindly notify us by return e-mail and discard and/or delete the communication. </font></td>
</tr>
<tr>
<td height="8" width="686"><font face="Arial" size="1">Thank you very much.</font></td>
</tr>
<tr>
<td height="8" width="686"><font face="Arial" size="1">It is possible for e-mails to be intercepted or affected by viruses. Whilst we maintain virus checks on our e-mails, we accept no liability for viruses </font></td>
</tr>
<tr>
<td height="8" width="686"><font face="Arial" size="1">or other material which might be introduced with this message.</font></td>
</tr>
</tbody></table><br><div class="MsoNormal" style="text-align: center;" align="center"><font face="Times New Roman" size="3"><span style="font-size: 12pt;">
<hr align="center" size="2" width="100%">
</span></font></div>
<p class="MsoNormal"><b><font face="Tahoma" size="2"><span style="font-size: 10pt; font-family: Tahoma; font-weight: bold;">Von:</span></font></b><font face="Tahoma" size="2"><span style="font-size: 10pt; font-family: Tahoma;"> Tom Downes
[mailto:<a href="mailto:tpdownes@caltech.edu" target="_blank">tpdownes@caltech.edu</a>] <br>
<b><span style="font-weight: bold;">Gesendet:</span></b> Montag, 19. April 2010
20:46<br>
<b><span style="font-weight: bold;">An:</span></b> <a href="mailto:kst@kde.org" target="_blank">kst@kde.org</a><br>
<b><span style="font-weight: bold;">Betreff:</span></b> [Kst] reading in NetCDF
matrix (2-dim vector)</span></font></p>
</div><div><div></div><div class="h5">
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size: 12pt;"> </span></font></p>
<div>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size: 12pt;">Hi:</span></font></p>
</div>
<div>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size: 12pt;"> </span></font></p>
</div>
<div>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size: 12pt;">I'm reasonably familiar with KST but am having a lot of trouble parsing
a NetCDF file in the fashion I intend. I am working on an instrument that
will have have around 100 independent detectors measured at 100 samples per
second each (i.e., around 10,000 total measurements per second). From the
perspective of data acquisition it is much easier to write to a NetCDF variable
with dimensions n_detectors x n_samples, where n_detectors would be this number
around 100 (126 in example below) and n_samples would equal 100 for 1 seconds
worth of data.</span></font></p>
</div>
<div>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size: 12pt;"> </span></font></p>
</div>
<div>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size: 12pt;">But trying to get KST to read this data in the way I intend seems to be
difficult. I want it to be able to split the matrix into 126 separate
vectors so that I can plot a timestream for each - realistically I would take a
representative sample. The "sample" and "frame"
terminology doesn't seem to quite match up to what I am doing - frames seem to
be more about bookkeeping than actually combining many timestream vectors into
a matrix.</span></font></p>
</div>
<div>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size: 12pt;"> </span></font></p>
</div>
<div>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size: 12pt;">Anyhow, I would like to solve this in a relatively simple way, with a
preference toward avoiding re-writing how the data is originally written to
disk as it is much simpler this way. Can I do what I need to do within
KST or with a simple plug-in? I am quite comfortable working in
C/C++/Java and shell scripting environments. Suggestions?</span></font></p>
</div>
<div>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size: 12pt;"> </span></font></p>
</div>
<div>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size: 12pt;">This is what the file looks like:</span></font></p>
</div>
<div>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size: 12pt;"> </span></font></p>
</div>
<div>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size: 12pt;">netcdf test-udp {</span></font></p>
</div>
<div>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size: 12pt;">dimensions:</span></font></p>
</div>
<div>
<p class="MsoNormal"><span><font face="Times New Roman" size="3"><span style="font-size: 12pt;"> </span></font></span>time
= UNLIMITED ; // (3960037 currently)</p>
</div>
<div>
<p class="MsoNormal"><span><font face="Times New Roman" size="3"><span style="font-size: 12pt;"> </span></font></span>nkids
= 126 ;</p>
</div>
<div>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size: 12pt;">variables:</span></font></p>
</div>
<div>
<p class="MsoNormal"><span><font face="Times New Roman" size="3"><span style="font-size: 12pt;"> </span></font></span>int
kids(time, nkids) ;</p>
</div>
<div>
<p class="MsoNormal"><span><font face="Times New Roman" size="3"><span style="font-size: 12pt;"> </span></font></span>kids:scale_factor
= 1 ;</p>
</div>
<div>
<p class="MsoNormal"><span><font face="Times New Roman" size="3"><span style="font-size: 12pt;"> </span></font></span>kids:add_offset
= 0 ;</p>
</div>
<div>
<p class="MsoNormal"><span><font face="Times New Roman" size="3"><span style="font-size: 12pt;"> </span></font></span>kids:units
= "bits" ;</p>
</div>
<div>
<p class="MsoNormal"><span><font face="Times New Roman" size="3"><span style="font-size: 12pt;"> </span></font></span>int
timestamp(time) ;</p>
</div>
<div>
<p class="MsoNormal"><span><font face="Times New Roman" size="3"><span style="font-size: 12pt;"> </span></font></span>timestamp:scale_factor
= 1 ;</p>
</div>
<div>
<p class="MsoNormal"><span><font face="Times New Roman" size="3"><span style="font-size: 12pt;"> </span></font></span>timestamp:add_offset
= 0 ;</p>
</div>
<div>
<p class="MsoNormal"><span><font face="Times New Roman" size="3"><span style="font-size: 12pt;"> </span></font></span>timestamp:units
= "bits" ;</p>
</div>
<div>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size: 12pt;">}</span></font></p>
</div>
<div>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size: 12pt;"> </span></font></p>
</div>
<div>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size: 12pt;">I am running KST 1.9.1 compiled manually after checking out the release
source. I don't think it matters, but I am running the kde3 libraries
alongside the kde4 libraries while logged into a KDE 3.5 WM session in OpenSUSE
11.2.</span></font></p>
</div>
<div>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size: 12pt;"> </span></font></p>
</div>
<div>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size: 12pt;">Tom</span></font></p>
</div>
</div></div></div>
</div>
</div>
</blockquote></div><br>