[Kst] reading in NetCDF matrix (2-dim vector)

Thu Apr 22 17:40:26 CEST 2010

I write the data as a matrix because I receive the data as UDP packets of
1xN_detectors every 10ms (100Hz sampling).  It seems more efficient to do 1
write-to-disk of N_detectors samples than N_detectors writes of 1 sample -
but I haven't actually tested this.  The existing analysis software assumes
a matrix so that is another reason to keep it that way.

I am working in Matlab which doesn't provide for user-accessible
multi-threading and therefore makes building a set of FIFO buffers more
difficult.  Matlab so far has proven to be able to handle being a UDP client
without losing data.  I'd rather not have to build a whole UDP client in C
or C++ where dividing the 1xN_detectors samples into N_detectors FIFO
buffers would be easier.

The only metadata I care about are scaling factors and offsets for
conversion to physical units, which are the same for all detectors.  I don't
think matrices are all that uncommon a way of doing things in NetCDF land -
half the point of NetCDF is to be able to expand the dimensions of your
matrices in a manner that is both efficient to store and to read.

KST 2b2 wouldn't even recognize a NetCDF file for what it was and I couldn't
get any of the SVN checkouts to compile.  So I installed KDE3 alongside KDE4
- works just fine.  All you need are the kde3 libraries - you can continue
to login using KDE4, if that's your preference.

I'll look at the datasource code and I'll be sure to share any work I do -
since the Matlab UDP client is working brilliantly, it may turn out to be
simpler to build a very stupid C program that just re-presents the data to
KST.

Thanks for getting back,

Tom

On Thu, Apr 22, 2010 at 7:17 AM, Brisset, Nicolas <
Nicolas.Brisset at eurocopter.com> wrote:

>  Hi Tom,
>
>
>
> the netcdf datasource for kst currently does not support reading in
> matrices. It should however be fairly easy to add if you feel so inclined,
> knowing that matrices are less well supported than vectors as far as I know.
> I myself have only worked with vectors up to now, so I may not be aware of
> everything. But I fear there is no easy way to slice matrices into vectors
> (). Plus it makes little sense to me, as each variable may have metadata and
> such attached to it and you would lose all that by grouping vectors into a
> matrix. That said, I don't know your constraints well enough. Maybe you have
> a good reason indeed to do it that way.
>
> Another path you may want to go is making some changes to the netcdf
> datasource so that when it finds matrices it provides their lines and
> columns as extra vectors. In that case all the work should be done at
> datasource level, but that definitely sounds feasible. You could query the
> dimensions of your 2-dimensional variables and add to the field list vectors
> like Matrix1/Column1, Matrix1/Column2, Matrix1/Line1, etc…
>
> Note: in the current code, it seems that the dimensions of netcdf vars are
> not checked, that is **bad**. For cdf, it is done better.
>
>
>
> Then, the other question you may ask yourself is whether to use kst 1.9.x
> or 2.x for that work. Kst 2.x is still beta quality, but starting to work
> and there have been quite a few changes (among others wrt datasources) so
> that what you do for one version of the netcdf datasource may require
> substantial porting to the other one. FYI, kst2 also works under Windows but
> nobody has yet tried netcdf under Windows. So it is still definitely
> experimental.
>
>
>
> Regards,
>
>
>
> Nicolas
>
>
>
>  *EUROCOPTER DEUTSCHLAND GmbH*  *
> Sitz der Gesellschaft / Registered Office: Donauwörth*  *
> Registergericht / Registration Court: Amtsgericht Augsburg HRB 16508*  *
> Vorsitzender des Aufsichtsrates / Chairman of the Supervisory Board: Dr. Lutz Bertling
> *  *Geschäftsführung / Board of Management:*  *Dr. Wolfgang Schoder, Vorsitzender / CEO; Friedrich-Wilhelm Hormel; Ralf
> Barnscheidt*         CONFIDENTIALITY NOTICE
> This communication and the information it contains is intended for the addressee ( s ) named above and for no other persons or organizations.
> It is confidential and may be legally privileged and protected by law. The unauthorized use, copying or disclosure of this communication
> or any part of it is prohibited and may be unlawful.
> If you have received this communication in error, kindly notify us by return e-mail and discard and/or delete the communication.
> Thank you very much.
> It is possible for e-mails to be intercepted or affected by viruses. Whilst we maintain virus checks on our e-mails, we accept no liability for viruses
> or other material which might be introduced with this message.
> ------------------------------
>
> *Von:* Tom Downes [mailto:tpdownes at caltech.edu]
> *Gesendet:* Montag, 19. April 2010 20:46
> *An:* kst at kde.org
> *Betreff:* [Kst] reading in NetCDF matrix (2-dim vector)
>
>
>
> Hi:
>
>
>
> I'm reasonably familiar with KST but am having a lot of trouble parsing a
> NetCDF file in the fashion I intend.  I am working on an instrument that
> will have have around 100 independent detectors measured at 100 samples per
> second each (i.e., around 10,000 total measurements per second).  From the
> perspective of data acquisition it is much easier to write to a NetCDF
> variable with dimensions n_detectors x n_samples, where n_detectors would be
> this number around 100 (126 in example below) and n_samples would equal 100
> for 1 seconds worth of data.
>
>
>
> But trying to get KST to read this data in the way I intend seems to be
> difficult.  I want it to be able to split the matrix into 126 separate
> vectors so that I can plot a timestream for each - realistically I would
> take a representative sample.  The "sample" and "frame" terminology doesn't
> seem to quite match up to what I am doing - frames seem to be more about
> bookkeeping than actually combining many timestream vectors into a matrix.
>
>
>
> Anyhow, I would like to solve this in a relatively simple way, with a
> preference toward avoiding re-writing how the data is originally written to
> disk as it is much simpler this way.  Can I do what I need to do within KST
> or with a simple plug-in?  I am quite comfortable working in C/C++/Java and
> shell scripting environments.  Suggestions?
>
>
>
> This is what the file looks like:
>
>
>
> netcdf test-udp {
>
> dimensions:
>
>           time = UNLIMITED ; // (3960037 currently)
>
>           nkids = 126 ;
>
> variables:
>
>           int kids(time, nkids) ;
>
>                       kids:scale_factor = 1 ;
>
>                       kids:add_offset = 0 ;
>
>                       kids:units = "bits" ;
>
>           int timestamp(time) ;
>
>                       timestamp:scale_factor = 1 ;
>
>                       timestamp:add_offset = 0 ;
>
>                       timestamp:units = "bits" ;
>
> }
>
>
>
> I am running KST 1.9.1 compiled manually after checking out the release
> source.  I don't think it matters, but I am running the kde3 libraries
> alongside the kde4 libraries while logged into a KDE 3.5 WM session in
> OpenSUSE 11.2.
>
>
>
> Tom
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.kde.org/pipermail/kst/attachments/20100422/1b34cf98/attachment-0001.htm 