[Kst] kst 2.x: dbus interface / datasource API

[D. V. Wiebe]

On Mon, Mar 08, 2010 at 05:07:06PM -0500, Matthew D Truch wrote:
> > Speaking of that, I was also wondering whether it would not be time to
> > look into getdata. We will certainly investigate a bit in that area,
> > though we were wondering how it performs if there are many variables
> > to be written so often into so many files. File IO has the reputation
> > of being slow. Is there some sort of buffering/memory caching being
> > done? Is there a limit on the number of variables that can be recorded
> > simultaneously? We routinely have simulations producing a few thousand
> > vars at 50 Hz... Maybe in such a case netCDF or CDF are better
> > choices?
> 
> I can't imaging that netCDF or CDF also don't write to disk?  But I've
> never used them in real time.
> 
> That said, I use getdata real time alot.  Our "standard" BLAST dataset
> has about 300 "variables" being written at 100 Hz along with an
> additional 600 variables being written at 5 Hz.  With very modest few
> year old machines, we can read and write these data with getdata with no
> problem at all.  I doubt that you'd have issues with your usage case.  
> 
> In fact, the reason we (Barth) came up with the "dirfile" that getdata
> reads/writes is because with that many variables, if you're only looking
> at a few at a time (even though you're writing them all), being able to
> scan forward and backward in the file is *greatly* improved over
> something that writes one large file that contains all the internal
> fields split up in time.  

You write data with GetData?  Or is there some metonymy going on here
between GetData (a software library) and Dirfile (a data format).

I admittedly haven't put it through it's paces, but my impression has
always been that the putdata() call in GetData isn't suited for
writing large quantities of data to dirfiles.  (Where "large quantity"
is any amount sufficiently large to make you concerned with things like
I/O speed.)  putdata() doesn't even use the rudimentary I/O buffering
provided by stdio.

-dvw
-- 
Don Wiebe                                   dvw at phas.ubc.ca
Department of Physics and Astronomy
University of British Columbia
6224 Agricultural Road                      Tele: +1-604-822-2585
University Endowment Lands, BC
Canada   V6T 1Z1                            http://ketiltrout.net/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
Url : http://mail.kde.org/pipermail/kst/attachments/20100308/e38e623f/attachment.sig