[Kst] Getdata library
Adam D. Hincks
ahincks at Princeton.EDU
Wed Aug 22 17:39:25 CEST 2007
This email is about the dirfile datasource used by KST; specifically, the
C library called getdata.
I work on the ACT project which uses dirfiles for its data storage. We
use KST for viewing data, and also are starting to use getdata for our
analysis pipeline. We prefer to use getdata (rather than write our
own) because it is "tested and true" and compatible with KST.
There are a some features not currently in the library that we have
decided we need, however, which I outline below. I am willing and able to
make these additions but of course would need to coordinate with whomever
is in charge of the source.
1. Byte-swapping. I have examined the source and it appears that there
are no provisions to do byte-swapping on differently-ended machines.
Right now we use all little-endian machines so this has not been an issue,
but we might do analysis in the future on architectures with different
byte orders. (Our files will all be recorded little-endian.) The
most convenient thing would be if getdata did byte-swapping somewhere in
the DoIfRaw() function.
2. A "truthful" version of GetNFrames(). GetNFrames() computes the
length of one of the files in the dirfile and uses that to compute the
number of frames. Then it subtracts 2. This is good when reading live
data as it avoids race conditions. But with finished data sets we will
always be losing 2 seconds of data (our frames are ~1 second long). We
would like a GetTrueNFrames().
3. "Ragged ends". This is a bit more complicated to explain, but here
Because of the nature of the ACT's data acquisition, we are unable to
guarantee that dirfiles will have a round number of frames. On our
faster channels there will inevitably be a spill-over after the last frame
of a number of samples. This number of samples will be the same for all
data at the same rate. So, for example, if we have fields at 10
samples per frame (SPF), 5 SPF and 1 SPF, we could have after the last frame:
10 SPF fields: 6 extra samples
5 SPF fields: 3 extra samples
1 SPF fields: 0 extra samples
The following would also be possible:
10 SPF fields: 3 extra samples
5 SPF fields: 2 extra samples
1 SPF fields: 1 extra sample
So we need a function to supplement GetNFrames(). This function would
return the number of extra samples after the last frame for a given field.
We can guarantee that data with the same SPF have the same number of extra
As I said before, I am able to make these changes. But we are reluctant
to have our own version of getdata and would prefer that the changes be
incorporated into the KST repository. Any help that you can provide would
be greatly appreciated and I can cooperate in any way to make the process
as expeditious as possible.
More information about the Kst