[rkward-devel] browser in rkward

Tue May 26 19:19:33 UTC 2009

Hi,

as I'm posting to rkward-devel, I'm leaving in all the context for others to 
pick up reading. 

On Tuesday 26 May 2009, you wrote:
> Thomas Friedrichsmeier wrote:http://dirk.eddelbuettel.com/code/rinside.htm
> > Hi,
> >
> > On Tuesday 26 May 2009, you wrote:
> > > I am interested in developping a front-end to R debugging abilities
> > > (browser, debug, ...), and rkward is a candidate host. This was a
> > > google summer of code idea for R this year, but was not retained
> > > eventually, so I will probably end up doing it on my own.
> >
> > I saw your mail on r-devel, and marked it as "I may want to look into
> > this some time", but I have to admit, I have not read the thread, yet.
> > Decent debugging support is certainly a much wanted feature in rkward.
> > If you would like to work on that, I'll try to be of help as much as I
> > can.
>
> Good. The more helpful you are, the more likely rkward is to be chosen
> as the first host for the feature, the more marketing you can do about
> it ;-)
>
> Although, I've already started to look into adding the feature to
> biocep, but this is not the first time I make proposal to the biocep
> team (karim) which are simply not taken care of, so ...
>
> > > Now, as it happens, the REPL implementation in rkward does not treat
> > > input from within the browser the same way as regular top level input,
> > > but rather brings a dialog box. This might be related to the use of
> > > R_ReplDLLdo1, instead of the usual REPL mechanism, but I am not sure (I
> > > have just been looking at the code today).
> >
> > The REPL-implementation in rkward is quite a mess. Not all of that is
> > the fault of R (some is simply bad code that I did not find the time
> > to improve, yet), but the basic problem is that the rkward-concept
> > just doesn't map well to a REPL-loop. The R REPL loop is basically
> > assuming one single channel of user input (namely a terminal).
>
> Yes, but if this leaves in its on thread, with a bit of care, this can
> survive without being the main thread. As an example JGR uses the REPL
> (through JRI). The relevant part of the readConsole callback is this:
>
>         output.append(prompt, JGRPrefs.CMD);
>             output.setCaretPosition(outputDoc.getLength());
>             String s = JGR.rSync.waitForNotification();
>             try {
>                 outputDoc.insertString(outputDoc.getLength(), s + "\n",
>                         JGRPrefs.CMD);
>             } catch (Exception e) {
>             }
>             return (s == null || s.length() == 0) ? "\n" : s + "\n";
>
> ... and most of the magic is in the JGR.rSync.waitForNotification()
> call, which basically sleeps until there is actually something to do. I
> am pretty sure this could be done in rkward as well.
>
> > In rkward, commands may originate from a number of different sources,
> > and may not necessarily be executed in the order that they are
> > generated (see RCommandStack, and the doxygen page "Using the
> > interface to R" (defined in rinterface.h), if you're interested in
> > some details).
>
> I'll have a look. This probably can survive alongside a proper REPL.
>
> > In much older versions of RKWard we did not use R's REPLs at all, but
> > basically relied on R_tryEval(). However, this approach has important
> > drawbacks as well. In rembedinternal.cpp around line 1000 you find a
> > rough explanation of how we try to tame the R REPL loop, and why. Also
> > see https://stat.ethz.ch/pipermail/r-devel/2007-January/044336.html
> > for some desiderata, if you like. I never got any feedback on that,
> > and did not pursue this any further.
> >
> >
> > (BTW, the reason we don't use the main REPL is that this never
> > returns. So that assumes that R is in charge of all event handling,
> > but unfortunately in rkward it's just the other way around.)
>
> again, not necessarily true, R just needs to believe it controls, but we
> can fool it.

Now that you mention it, I think you're right, and in fact I've played with 
that idea a bit in the past. techdocs/r_eventloops picks it up in the context 
of another long-term project (which is to split out the R backend into a 
separate process, which could even run on a remote machine).

It's simply been a while since I last thought about these issues.

That said, while restructuring the backend to run the regular R REPL, may be 
worth-while, I don't think it really makes much of a difference, in this 
context, or if so, I still fail to see, how.

> > In a nutshell, the issue with the REPL is that R_ReadConsole is used
> > for several quite distinct purposes:
> > 1) Waiting for toplevel - parsable/command - input.
> > 2) Fetching more of the toplevel input, if it was too long to fit in
> > the buffer.
> > 3) Requests for generic user input -> readline().
> > 4) The browser.
> > In rkward we need to spend quite some effort on finding out whether
> > R_ReadConsole is doing 1, 2 or 3/4, as that info is not readily
> > available. 3 and 4 are not yet differentiated, and both treated the
> > same in rkward (try readline("test"); to see it's actually the same
> > dialog).
> >
> >
> > So the first task would be to find out, when exactly a call to
> > R_ReadConsole signifies we are in a debugging-context (4) instead of a
> > generic input request (3). Probably this is not too hard to do, but I
> > simply haven't looked into that, yet.
>
> The prompt argument of the readConsole callback gives you a clue I
> suppose, but the key I think (from what simon urbanek told me on the
> R-devel thread) is that if you use the REPL, you don't have to care so
> much.

If I understood that correctly, he's basically just saying that R_ReadConsole 
gets called from inside the REPL when in the browser(). The same thing is true 
in our version of the loop.

In fact, as long as we don't return from the toplevel statement, there should 
be (almost) no difference between running the full loop and R_ReplDLLdo1. The 
main difference is that Rf_mainloop continues after returning from a toplevel 
statement, while R_ReplDLLdo1 does not. When debugging we stay inside a 
statement all the time. Note that I'm not oppossed to switching to the full 
REPL, at all. Just saying that perhaps it's not a requirement, here.

Yes, probably the prompt argument is all the cues we need to find out were in 
the browser(). So we can just go and feed it the commands we want it to run...

> > The second task would be to signal that to the main/GUI thread.
> > RThread::handleStandardCallback() or RThread::handleSubstackCall()
> > could be used for that. This would then allow a specialized debugger
> > widget/window (task number three) to run further R commands in the
> > current context to gather relevant information such as the call stack
> > / code of the function we're in, .... (Here we have another small -
> > but solvable - issue that currently rkward evaluates all statements in
> > the globalenv()).
> >
> >
> > So all in all, adding a nice debugger should be possible without
> > having to change the whole application design, but of course it's not
> > exactly trivial, either.
> >
> >
> > Let me know, what I can do to help you with this.
>
> My guesss is that the first step is going to be to set up an actual
> REPL, with all the pain this can cause. I can look into it although I am
> not so familiar with Qt and KDE, but the code looks alright with enough
> comments to pick things up.

Some more technical comments on this:
- Historically we could not bring Qt-includes and R-includes together in the 
same source file. This is the reason, why rthread.cpp and rembedinternal.cpp 
are separated the way they are. It may - or may not - be helpful to merge the 
two.
- Partially for the same reason, some of the data formats are quite strange. 
I'm quite embarassed of RCallbackArgs, for example, and hope to change it one 
day.
- We have a lot of legacy support for ancient versions of R. I intend to drop 
everything before R 2.7.x in trunk. This might simplify the backend code quite 
a bit in a few places.

Regarding this, how do you think you will proceed? If you're going to start 
playing with the backend code right away, I'll simply refrain from doing any 
non-needed changes for the time being. In contrast, if you're going to spend a 
few more weeks on the conceptual level, it might be quite helpful if I start 
working on some cleanups as soon as possible.

> Have you looked into Dirk's Rinside:
> http://dirk.eddelbuettel.com/code/rinside.html

No, not yet, and ATM the server just gives me an error.

> Do you think this discussion should be broadcast to one of your mailing
> list ? which one ?

Yes, in fact. I'm taking this to rkward-devel.

Regards
Thomas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.kde.org/pipermail/rkward-devel/attachments/20090526/eb749250/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.kde.org/pipermail/rkward-devel/attachments/20090526/eb749250/attachment.sig>