[Kst] Real-time data display without files ?

[Dirk Eddelbuettel]

Matt,

On 23 April 2006 at 10:05, Matthew D Truch wrote:
| > Following on my earlier post, I have a more generic design question:
| > 
| >     Can I build a plugin that 'just listens' and gets data from 
| >     some other callback mechanism -- without requiring a file?  
| > 
| > The answer is probably provided in a thread from September 2004
| > 
| >     http://mail.kde.org/pipermail/kst/2004-September/003540.html
| >     http://mail.kde.org/pipermail/kst/2004-September/003555.html
| > 
| > and may simply be 'nope' but I thought I'd ask anyway.  
| 
| Yes, you can build your own plugin to do that; however, none of the
| *existing* plugins shipped with kst do that.  Eventually (perhaps quite

Yes, but how would I get it loaded / started? I saw no command-line or menu
option for it ...  Can you give me a pointer as to where to start digging?

| quickly) your datasource would consume a large amount of memory to hold
| the past values that you are reading (until a restart of kst of
| what-have-you), but that's really your only obstacle.  

Just to be sure: isn't that the same with file-based data? If kst read 40mb,
it 'has' 40mb. 

It shouldn't be a problem, our data tends to in 'skinny vector' form: lots of
rows, few columns. If it grows to a few dozen mb it is still way less than my
emacs or firefox session :)

| > My data can come in at maybe 200 to 300 updates per second at peak. I also

I forgot to stress that the median is more like 10 to 20 to 30 updates a
second. Which should be doable. During peaks we can easily drop a few
updates.  So the file-based approach has a lot going for it in terms of
testing and experimentation.

And while out running earlier this morning, it occurred to me that kst, with
the dual focus on the command-line, is probably well behaved in the Unix
tradition and takes takes stdin as a filename -- and indeed, the following
simple R/shell script generates suitable data with C times and random
gaussian noise:

edd at basebud:~> cat /tmp/kstdemo.sh
#!/bin/sh

cat <<EOF | R --slave
options('digits'=14)
options('digits.secs'=4)
while(1) {
   cat(as.numeric(Sys.time()), 100+3*rnorm(1),'\n')
   Sys.sleep(0.005)
}
EOF

[ options() sets the needed display granularity for the C time, and activates
  the display of milli seconds -- that last feature needs R 2.3.0, due
  tomorrow, or any of the recent pre-releases -- and we add a 5 millisecond
  sleep to not drown kst ]

which we can pipe into kst via a simple

edd at basebud:~> /tmp/kstdemo.sh | kst -x 1 -y 2 -   

Very sleek!!  All that is left is to tell kst to take X values as C time and
to show them in a suitable format, as you kindly reminded me yesterday.

So all I have to do at work is to replace kstdemo.sh with a command-line tool
that listen to our data --- easy as I just wrote one of those for other
purposes.  Very nice -- pluggable Unix toolchains save the day.

| > may want to view and relate several such streams.  As a start, I should be
| > able to plug into that data pretty easily via code that listens to our data
| > streams and emits csv data (for which I may just have to reformat the time
| > stamp to get suitable 'ctime' seconds for kst).
| > 
| > However, for efficiency, could I just 'listen' to data that is not coming via
| > files -- without re-architecting kst?  It just doesn't seem right to divert
| > all that data to file only to read it back later.  (I should note that we
| > already have existing logging and capturing mechanisms.)
| 
| As you state, you already log and capture your data.  The 'standard' kst
| (as in it's how kst was originally intended) way of doing things is to
| have kst read the data from the logs.  It is true that you are diverting
| "all of that data to file only to read it back later," but 'later' may
| be milliseconds, not seconds (or longer), provided the logger flushes

It may matter how many milliseconds I am behind. Shaving the 'write to file,
read from file' cycle seems like an obvious improvement. Stdin is a good
alternative as we're down to 'format/print, read, parse'. Passing binary data
around could improve this. I'll have to see if I need it.

| the file it's writing at regular (as fast as you want) intervals.  As
| long as kst can read the type of file that the 'logger' generates, kst
| can plot the data 'real time' with new data appearing on the plot 'as
| soon' as it is captured and written by the logger, where 'as soon' is
| defined as with an on order 100 milliseconds delay.

I take it the file reader is 'smart' and has a file pointer to not re-read
everything from the beginning?  Or is that handled inside kst via the
KstObject::UPDATE vs KstObject::NO_CHANGE and the plugin has to figure out
how to be efficient?

| Of course, if you don't want to log all data, then this forces you to
| log some data you don't need, but you can easily delete it at a later
| time.  However, versus the method where the kst datasource reads the
| captured data itself into memory, it gives you the added benefit that
| you may look at data that occurred before that kst instance was started,
| including just before.  
| 
| Whichever route you choose, let us know how you're doing, and we'll help
| out wherever we can.  Good luck. 

Thanks a bunch -- you've already been very helpful.

Regards, Dirk

-- 
Hell, there are no rules here - we're trying to accomplish something. 
                                                  -- Thomas A. Edison