[Kst] getData flexibility

[Theodore Kisner]

My opinion:  I like it :-)  It's a way of having a huge virtual vector where 
memory is only allocated for the elements that exist.

I can think of other future applications.  Imagine we someday have a healpix 
KstBaseCurve that reads a vector of pixel values from the healpix datasource.

Now imagine that the healpix FITS file we are reading covers the full sky and 
is hundreds of MB.  We really only want to read a portion of the sky, so in 
the datasource config, we specify an RA/DEC range.  The vector that gets 
returned by the datasource contains only those pixels within that region of 
the sky, and the vector itself keeps track of the pixel index ranges that are 
available.

Of course, this would require slight future rework of the datasources (so that 
we are not just writing data to a double*), but that is likely going to 
happen anyway.

just my $0.02

-Ted


On Wednesday 03 May 2006 11:50, Eli Fidler wrote:
| George and I have discussed this issue some more. I think that we need to
| change the internals of Kst, particularly RVector. I think that instead of
| containing a range of samples from frame 0 to n, we should be able to store
| a set of frame ranges.
|
| Some situations where this is useful:
| 1. The cache contains frames 10-20 and 100-500 of a field. The server is
| unreachable. Kst can use the stored frames from the cache and add the other
| frames once the server is reachable. Without frame ranges, I would have to
| either say the datasource is empty, allow the user to use the first range
| only or allow the user to retrieve all the frames but 0-9 and 21-99 will be
| filled with NaNs. None of these solutions are very good.
|
| 2. Progressive loading. The user could retrieve frames 0-end skip 10, then
| 1-end skip 10, then 2-end skip 10... The RVector would know which frames
| are missing and still need to be retrieved.
|
| 3. I think frame ranges is a better representation of holes in the data
| than NaNs. It could be useful to visualize missing frame ranges by shading
| in the gui. Frame ranges would work for masking as well.
|
| 4. We could remove the hack in the piolib datasource which moves all the
| vectors to frame 0. I don't know exactly what the dirfile datasource does
| with data that doesn't begin at frame 0 as I don't have such a dirfile, but
| a similar hack likely exists.
|
| I have a class in NAD which stores sets of frame ranges (QPair<uint32_t,
| uint32_t>'s actually). It allows you to add ranges, intersect ranges, skip,
| etc. I think it could be used in Kst with some modifications. The header
| file is attached.
|
| I'm interested in people's thoughts on this idea.
|
| Eli
|
| On Tuesday 02 May 2006 18:08, Eli Fidler wrote:
| > I'm in the process of writing the new NAD KstDataSource with uses the
| > local cache. I've run into a case of getData() where the current
| > interface seems insufficient.
| >
| > When using the NAD cache, I may have frames 1 to 50 of a particular field
| > in the cache. If the NAD server connection is unavailable, what should I
| > do if the user requests frames 1 to 100? What about 1 to 50?
| >
| > I assume I should just give them the data in the second case and pretend
| > everything is fine. In the first case, should I return partial data? no
| > data? something else?
| >
| > What if I have part of the data, but it's not the first part?
| >
| > The problem stems from the fact that I can only return -1 for any error,
| > so I can't signal a partial success very easily.
| >
| > Eli
| > _______________________________________________
| > Kst mailing list
| > Kst at kde.org
| > https://mail.kde.org/mailman/listinfo/kst