[Kst] Getdata library

[Theodore Kisner]

You might also consider a slightly modified dirfile library that I created 
here:

http://cmb.phys.cwru.edu/kisner/code/dirfile/

it includes byte-swapping (all datafiles are stored on disk as little-endian, 
even on big-endian machines), and "putData" functionality.  It also was not 
meant to necessarily be used with kst, so each field is treated as having an 
independent number of frames.  This means that there is no mechanism 
for "synchronizing" fields.  

For example, in kst, each field in a given dirfile has the same number of 
frames, and a "master" field is checked to get this number of frames.  When 
writing data in real time, this "master" field is updated last, which 
guarantees that ALL fields have at least as many frames as the master.

In libdirfile, none of this is true- a dirfile is just a collection of binary 
files, each with an arbitrary length.

One recent change added was a set of functions to explicitly close the 
filehandles associated with the fields.  For performance, the filehandles are 
cached in a global structure, but on some systems there are problems having 
more than 1024 files open at once.

Also, I tried to keep Barth's original coding style when I forked the code 
several years ago.  If I was doing it over from scratch at this point, there 
are definitely some things I would clean up.  

Hope this is useful...

-Ted


On Wednesday 22 August 2007 08:39, Adam D. Hincks wrote:
> Dear KSTers,
>
> This email is about the dirfile datasource used by KST; specifically, the
> C library called getdata.
>
> I work on the ACT project which uses dirfiles for its data storage.  We
> use KST for viewing data, and also are starting to use getdata for our
> analysis pipeline.  We prefer to use getdata (rather than write our
> own) because it is "tested and true" and compatible with KST.
>
> There are a some features not currently in the library that we have
> decided we need, however, which I outline below.  I am willing and able to
> make these additions but of course would need to coordinate with whomever
> is in charge of the source.
>
> 1.  Byte-swapping.  I have examined the source and it appears that there
> are no provisions to do byte-swapping on differently-ended machines.
> Right now we use all little-endian machines so this has not been an issue,
> but we might do analysis in the future on architectures with different
> byte orders.  (Our files will all be recorded little-endian.)  The
> most convenient thing would be if getdata did byte-swapping somewhere in
> the DoIfRaw() function.
>
> 2.  A "truthful" version of GetNFrames().  GetNFrames() computes the
> length of one of the files in the dirfile and uses that to compute the
> number of frames.  Then it subtracts 2.  This is good when reading live
> data as it avoids race conditions.  But with finished data sets we will
> always be losing 2 seconds of data (our frames are ~1 second long).  We
> would like a GetTrueNFrames().
>
> 3.  "Ragged ends".  This is a bit more complicated to explain, but here
> goes:
>
> Because of the nature of the ACT's data acquisition, we are unable to
> guarantee that dirfiles will have a round number of frames.  On our
> faster channels there will inevitably be a spill-over after the last frame
> of a number of samples.  This number of samples will be the same for all
> data at the same rate.  So, for example, if we have fields at 10
> samples per frame (SPF), 5 SPF and 1 SPF, we could have after the last
> frame:
>
>     10 SPF fields:  6 extra samples
>     5 SPF fields:   3 extra samples
>     1 SPF fields:   0 extra samples
>
> The following would also be possible:
>
>     10 SPF fields:  3 extra samples
>     5 SPF fields:   2 extra samples
>     1 SPF fields:   1 extra sample
>
> So we need a function to supplement GetNFrames().  This function would
> return the number of extra samples after the last frame for a given field.
> We can guarantee that data with the same SPF have the same number of extra
> samples.
>
>
> As I said before, I am able to make these changes.  But we are reluctant
> to have our own version of getdata and would prefer that the changes be
> incorporated into the KST repository.  Any help that you can provide would
> be greatly appreciated and I can cooperate in any way to make the process
> as expeditious as possible.
>
> Thanks,
>
> Adam
> _______________________________________________
> Kst mailing list
> Kst at kde.org
> https://mail.kde.org/mailman/listinfo/kst