[Kst] Re: CDF datasource debugging ?

[Nicolas.Brisset at eurocopter.com]

On Thursday 15 July 2004 22:21, George Staikos wrote:
> I have the CDF plugin working here now, with a sample database.  So far it
> looks to be ok - it just needs some bug fixes.  
Reading from CDF is not so complicated that talented and experimented 
programmers like you could not handle this within a reasonable time :-)

> I don't fully understand how CDF works though, so I can't comment too much 
> on how it fits into Kst yet.
CDF is hard to understand at the beginning due to the specific terminology 
they use (and if you look at the source code of their curses-based tools, be 
prepared : this is not nice well structured C++ !! By the way, I found kscope 
very handy to examine this code: this piece of code is really a great tool 
for such cases where you need to get to grips with some external code).

Here is a CDF quickstart guide (the docs included in the distribution explain 
that fairly well):
- in CDF, variables can have different dimensionalities, which means that for 
each sample (called "record") you may have a scalar (dimension 0), a vector 
(dimension 1), a matrix (dimension 2), etc... In my code I deliberately chose 
to use only 0-dimensional variables. I suppose this is sensible for a 2D 
plotting tool...
- the number of records may differ from one variable to the next
- the storage uses a very compact binary platform-independent (no 
big/little-endian problems) format
- there can be "attributes" attached globally to the file or to each variable 
with arbitrary metadata in them
- for historical reasons, there can be 2 types of variables in a cdf file, 
called rVars and zVars (zVars superseding rVars). In the files I have, there 
are only rVars but I tried to use the same macros they use in the official 
tools to handle both transparently (rVars may be lagging behind right now, 
though)
- to look at the contents of a cdf file, the handiest way is to use the 
"cdfexport" tool. A bit rough for people used to nice Qt/KDE widgets :-) but 
still pretty efficient when you know the basics:
	1) load the cdf file with "cdfexport file" when the file is called file.cdf, 
which is not really intuitive !
	2) you will get a list of variables and type/dimension information. From 
there, you can look at the options in the "popup" brought by Ctrl+A, 
especially 'list to file...' which will create an ASCII file. You can select 
which variables to export first, which you should do in case you have vectors 
or matrices (i.e. vars with dimensions > 0) otherwise the ASCII file will be 
hard to use and get very big. Beware, the sample I sent includes some such 
vars towards the end of the list !

Now, one *IMPORTANT POINT* regarding the organization in "blocks" I mentioned 
in my post yesterday, which Barth suggested to encode into kst: this is NOT a 
CDF standard. It just happens that the files I work with use this 
_convention_ between the people who create them and us. For that reason I 
would not recommend to hard-code the notion of groups into the kst 
datasource, which in my view should be generic and work with any cdf file out 
there. That's why I prefer to indicate a reading problem when asked for a 
non-existing value. It is however not clear to me what the best way to 
achieve this is: return a lot of NaNs in the v buffer (the CDFlib will return 
0 when a record beyond the last available for a variable is queried, but we 
can store NaNs in v), or stop at the last available data and use the returned 
int to indicate the number of values read (possibly lower than the argument 
n, or 0/-1 if none) ? The second looks nicer to me, but I do not know the 
internals of kst.

> In piolib we have a gross hack that needs to be fixed.  Actually this 
> is high on my TODO list.  Please don't look at that code. :-)
You are tempting me :-)

Thanks to all for your kind help. Feel free to ask for more information in 
case something is still unclear.

Nicolas