[RkWard-devel] [rkward - Help] RE: Large matrices

[SourceForge.net]

Read and respond to this message at: 
https://sourceforge.net/forum/message.php?msg_id=4159536
By: tfry

PK already explained why this doesn't work. It is a memory issue.

Just some additional thoughts:
1) Keep in mind, that R may need to duplicate the entire object. This typically
happens when writing to it (as far as I understand, it may also happen in a
few other circumstances, but R tries to avoid unnecessary copies). Hence, even
if the object is larger than half the available RAM, you are quite likely to
get into trouble.

2) RKWard does impose a memory overhead, but that should not be significant
in this case (the overhead is mostly constant, until you start editing the object
(which is not yet supported for matrices, anyway)).

3) Since R will copy objects on write, initializing a very large object from
a for loop is probably very inefficient. For each iteration (or at least for
each assignment, even to a single cell in the matrix), the entire object would
be copied (as far as I understand, maybe there are some low-level optimizations
in R for this case). At least you should make sure to assign in large chunks
to minimize the total number of assignment to the matrix. One alternative might
be to first generate the data in a file, and then to read that file - or of
course program it in C.

4) I don't know the nature of your data, or what you intend to use it for. But
maybe you could trade off speed for memory by generating all data on the fly.
I.e., you'd create a function get.data.value (row, col) that calculates a single
cell on the fly (or perhaps a single row), and returns it. It may even be possible
to create a matrix-like object with matrix like subsetting operators that generates
the data on demand. However, please don't ask me about details, here.

Maybe some of this helps...

______________________________________________________________________
You are receiving this email because you elected to monitor this forum.
To stop monitoring this forum, login to SourceForge.net and visit: 
https://sourceforge.net/forum/unmonitor.php?forum_id=165574