[Kst] Does kst have any support for data that changes, but doesn't grow?

[D. V. Wiebe]

On Thu, Dec 12, 2013 at 05:01:14PM +0000, Fowler, Joseph W. wrote:
> Dear Barth,
> 
> Thanks for giving it some thought. You're right: the writer (or reader) would have to lock the file somehow. I hadn't thought of that.
> 
> In my understanding of GetData and kst, though, the real issue is that the idea of altering a data source rather than appending to it violates the basic abstraction of what a data source is. That's why it makes sense for kst not to read old data.
> 
> For the sake of argument, I picture a system where the a field can be tagged as volatile. A volatile field means that (1) it can change data without changing its length, and (2) its length is independent of the number of frames in the reference field. In dirfiles, I don't know if you'd want to have a /VOLATILE directive, or let VOLATILE be a basic type, an alternative to RAW. Probably the latter, since "samples per frame" makes no sense for volatile fields. Derived fields would inherit volatility from the raw one.
> 
> It might be too much of a departure and create too many problems. For example, what's the implied INDEX vector if a user wants to plot a volatile field? Will it depend on whether the volatile field is longer or shorter than the number of frames in the reference field? Also, how will kst know when to check the volatile field for changes?
> 
> Still, it might be worth it for you to consider it. You can see from the thread http://mail.kde.org/pipermail/kst/2013-August/021394.html that I'm not the first this year to ask about achieving oscilloscope-type behavior from kst.
> 
> For my immediate problem, I figure I'll either try to write a kst plugin for our obscure data file format or get our group to agree to using dirfiles for our raw data.
> 
> Thanks.
> Joe

Hey Joe,

You seem to be asking for two separate things here:

1) A way to have GetData inform kst when data which kst has already read
have changed.

2) Have non-synchronously sampled data in a dirfile.

From the standpoint of GetData, the first of these is probably the easier
to accomodate, but as Barth points out it would require some sort of
handshake between reader and writer to avoid race conditions.

The second requires a more fundamental change to the Dirfile Standard,
and the way GetData is structured, but may be possible.

I'll think about what can be done about it.

Not dirfile related, but I recall people talking about how to get kst to
read data files which only present a single value which is repeatedly
overwritten (for example, various /proc files), but I don't know if
anything happened with that.

Cheers,
-dvw

Addendum:

Hmmm, I hesitate to mention this, because it's probably not worth the
effort, is sub-optimal in implementation, and I don't think kst
supports it, but, after thinking about it, there *is* a way to implement
"oscilliscope mode" in a Dirfile as-is.

Here's the jist of it:

On the writer side, instead of storing the data in a binary RAW field
file, put the oscilliscope data in a CARRAY scalar constant array in
the Dirfile metadata (the format file), and periodically flush the
metadata to disk to update it.  The downside is that CARRAY data are
printf'd as formatted text to the format file, meaning writes will be
slow; also, if amount of data is variable, or if the whole trace
isn't updated at once, things are more annoying:

  extern size_t trace_len; /* intialised on dirfile open */

  void write_trace(DIRFILE *D, const double *data, size_t npts)
  {
    if (npts != trace_len) { /* resize CARRAY */
      gd_alter_carray(D, "trace", GD_NULL, npts);
      trace_len = npts;
    }

    /* update the whole trace -- use gd_put_carray_slice to only
        update a portion of the CARRAY */
    gd_put_carray(D, "trace", GD_FLOAT64, data);

    /* commit */
    gd_metaflush(D);
  }

On the reader side, use gd_desync (or something similar), to monitor
the dirfile for metdata updates and re-fetch the CARRAY when a change
is detected.  Downsides here include that re-reading the CARRAY from
disk requires closing and re-opening the dirfile (The "C" in CARRAY
stands for "CONST").  Also, gd_desync is somewhat lame (it uses mtime
to check for changes to the format file, maning it typically can't
detect metadata updates at a rate of more than 1 Hz).

WRT kst, AFAIK, it doesn't check for metadata updates (via gd_desync
or any other method) nor did it read CARRAYs last time I checked.  (It
may ignore top-level scalar metdata completely.)

  /* returns zero if the data haven't changed, or the number of points
     read */
  size_t read_trace(DIRFILE *D, double *data_out)
  {
    /* check for updated metadata on disk (and reload the dirfile
       if needed) */
    int update = gd_desync(D, GD_DESYNC_REOPEN);

    if (update == 0) {
      return 0; /* no new data */
    } else if (update == -1) {
      /* error ... */
    }

    /* record length */
    size_t trace_len = gd_carray_len(D, "trace");

    /* do some bounds checking on data_out here ... */

    /* fetch */
    gd_get_carray(D, "trace", GD_FLOAT64, data_out);

    return trace_len;
  }

Faster update detection rates could be achieved with a better
replacement to gd_desync. With the CARRAY in a /INCLUDEd
file, using gd_uninclude/gd_include to force GetData to re-read
that file might be less expensive than re-opening the whole
dirfile, but it's still not going to be great.
-- 
Don Wiebe                                   dvw at phas.ubc.ca
Department of Physics and Astronomy
University of British Columbia              Hennings 204
6224 Agricultural Road                      Tele: +1-604-822-2585
University Endowment Lands, BC
Canada   V6T 1Z1                            http://ketiltrout.net/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
URL: <http://mail.kde.org/pipermail/kst/attachments/20131213/7b4c0046/attachment.sig>