[rkward-devel] browser in rkward

[Romain Francois]

Thomas Friedrichsmeier wrote:
> Hi,
>
>
> as I'm posting to rkward-devel, I'm leaving in all the context for 
> others to pick up reading.
>
>
> On Tuesday 26 May 2009, you wrote:
> > Thomas Friedrichsmeier 
> wrote:http://dirk.eddelbuettel.com/code/rinside.htm
> > > Hi,
> > >
> > > On Tuesday 26 May 2009, you wrote:
> > > > I am interested in developping a front-end to R debugging abilities
> > > > (browser, debug, ...), and rkward is a candidate host. This was a
> > > > google summer of code idea for R this year, but was not retained
> > > > eventually, so I will probably end up doing it on my own.
> > >
> > > I saw your mail on r-devel, and marked it as "I may want to look into
> > > this some time", but I have to admit, I have not read the thread, yet.
> > > Decent debugging support is certainly a much wanted feature in rkward.
> > > If you would like to work on that, I'll try to be of help as much as I
> > > can.
> >
> > Good. The more helpful you are, the more likely rkward is to be chosen
> > as the first host for the feature, the more marketing you can do about
> > it ;-)
> >
> >
> > > > Now, as it happens, the REPL implementation in rkward does not treat
> > > > input from within the browser the same way as regular top level 
> input,
> > > > but rather brings a dialog box. This might be related to the use of
> > > > R_ReplDLLdo1, instead of the usual REPL mechanism, but I am not 
> sure (I
> > > > have just been looking at the code today).
> > >
> > > The REPL-implementation in rkward is quite a mess. Not all of that is
> > > the fault of R (some is simply bad code that I did not find the time
> > > to improve, yet), but the basic problem is that the rkward-concept
> > > just doesn't map well to a REPL-loop. The R REPL loop is basically
> > > assuming one single channel of user input (namely a terminal).
> >
> > Yes, but if this leaves in its on thread, with a bit of care, this can
> > survive without being the main thread. As an example JGR uses the REPL
> > (through JRI). The relevant part of the readConsole callback is this:
> >
> > output.append(prompt, JGRPrefs.CMD);
> > output.setCaretPosition(outputDoc.getLength());
> > String s = JGR.rSync.waitForNotification();
> > try {
> > outputDoc.insertString(outputDoc.getLength(), s + "\n",
> > JGRPrefs.CMD);
> > } catch (Exception e) {
> > }
> > return (s == null || s.length() == 0) ? "\n" : s + "\n";
> >
> > ... and most of the magic is in the JGR.rSync.waitForNotification()
> > call, which basically sleeps until there is actually something to do. I
> > am pretty sure this could be done in rkward as well.
> >
> > > In rkward, commands may originate from a number of different sources,
> > > and may not necessarily be executed in the order that they are
> > > generated (see RCommandStack, and the doxygen page "Using the
> > > interface to R" (defined in rinterface.h), if you're interested in
> > > some details).
> >
> > I'll have a look. This probably can survive alongside a proper REPL.
> >
> > > In much older versions of RKWard we did not use R's REPLs at all, but
> > > basically relied on R_tryEval(). However, this approach has important
> > > drawbacks as well. In rembedinternal.cpp around line 1000 you find a
> > > rough explanation of how we try to tame the R REPL loop, and why. Also
> > > see https://stat.ethz.ch/pipermail/r-devel/2007-January/044336.html
> > > for some desiderata, if you like. I never got any feedback on that,
> > > and did not pursue this any further.
> > >
> > >
> > > (BTW, the reason we don't use the main REPL is that this never
> > > returns. So that assumes that R is in charge of all event handling,
> > > but unfortunately in rkward it's just the other way around.)
> >
> > again, not necessarily true, R just needs to believe it controls, but we
> > can fool it.
>
>
> Now that you mention it, I think you're right, and in fact I've played 
> with that idea a bit in the past. techdocs/r_eventloops picks it up in 
> the context of another long-term project (which is to split out the R 
> backend into a separate process, which could even run on a remote 
> machine).
>
>
> It's simply been a while since I last thought about these issues.
>
>
> That said, while restructuring the backend to run the regular R REPL, 
> may be worth-while, I don't think it really makes much of a 
> difference, in this context, or if so, I still fail to see, how.

You might be right, I need to do more checking. As I said, I've just
started to read the code today.

>
> > > In a nutshell, the issue with the REPL is that R_ReadConsole is used
> > > for several quite distinct purposes:
> > > 1) Waiting for toplevel - parsable/command - input.
> > > 2) Fetching more of the toplevel input, if it was too long to fit in
> > > the buffer.
> > > 3) Requests for generic user input -> readline().
> > > 4) The browser.
> > > In rkward we need to spend quite some effort on finding out whether
> > > R_ReadConsole is doing 1, 2 or 3/4, as that info is not readily
> > > available. 3 and 4 are not yet differentiated, and both treated the
> > > same in rkward (try readline("test"); to see it's actually the same
> > > dialog).
> > >
> > >
> > > So the first task would be to find out, when exactly a call to
> > > R_ReadConsole signifies we are in a debugging-context (4) instead of a
> > > generic input request (3). Probably this is not too hard to do, but I
> > > simply haven't looked into that, yet.
> >
> > The prompt argument of the readConsole callback gives you a clue I
> > suppose, but the key I think (from what simon urbanek told me on the
> > R-devel thread) is that if you use the REPL, you don't have to care so
> > much.
>
>
> If I understood that correctly, he's basically just saying that 
> R_ReadConsole gets called from inside the REPL when in the browser(). 
> The same thing is true in our version of the loop.

What confuses me is that R_ReadConsole is __not__ used by R to request
top-level commands, hence there is a difference between "within the
browser" and "outside"

> In fact, as long as we don't return from the toplevel statement, there 
> should be (almost) no difference between running the full loop and 
> R_ReplDLLdo1. The main difference is that Rf_mainloop continues after 
> returning from a toplevel statement, while R_ReplDLLdo1 does not. When 
> debugging we stay inside a statement all the time. Note that I'm not 
> oppossed to switching to the full REPL, at all. Just saying that 
> perhaps it's not a requirement, here.
>
>
> Yes, probably the prompt argument is all the cues we need to find out 
> were in the browser(). So we can just go and feed it the commands we 
> want it to run...
The catch here is that you might have statements over multiple lines
inside the browser, and in that case the regular continue prompt is used.

> > > The second task would be to signal that to the main/GUI thread.
> > > RThread::handleStandardCallback() or RThread::handleSubstackCall()
> > > could be used for that. This would then allow a specialized debugger
> > > widget/window (task number three) to run further R commands in the
> > > current context to gather relevant information such as the call stack
> > > / code of the function we're in, .... (Here we have another small -
> > > but solvable - issue that currently rkward evaluates all statements in
> > > the globalenv()).
> > >
> > >
> > > So all in all, adding a nice debugger should be possible without
> > > having to change the whole application design, but of course it's not
> > > exactly trivial, either.
> > >
> > >
> > > Let me know, what I can do to help you with this.
> >
> > My guesss is that the first step is going to be to set up an actual
> > REPL, with all the pain this can cause. I can look into it although I am
> > not so familiar with Qt and KDE, but the code looks alright with enough
> > comments to pick things up.
>
>
> Some more technical comments on this:
> - Historically we could not bring Qt-includes and R-includes together 
> in the same source file. This is the reason, why rthread.cpp and 
> rembedinternal.cpp are separated the way they are. It may - or may not 
> - be helpful to merge the two.
> - Partially for the same reason, some of the data formats are quite 
> strange. I'm quite embarassed of RCallbackArgs, for example, and hope 
> to change it one day.
> - We have a lot of legacy support for ancient versions of R. I intend 
> to drop everything before R 2.7.x in trunk. This might simplify the 
> backend code quite a bit in a few places.
>
>
> Regarding this, how do you think you will proceed? If you're going to 
> start playing with the backend code right away, I'll simply refrain 
> from doing any non-needed changes for the time being. In contrast, if 
> you're going to spend a few more weeks on the conceptual level, it 
> might be quite helpful if I start working on some cleanups as soon as 
> possible.

It is probably better if you proceed first with the modifications you
have in mind.

>
> > Have you looked into Dirk's Rinside:
> > http://dirk.eddelbuettel.com/code/rinside.html
>
>
> No, not yet, and ATM the server just gives me an error.

See also:
- http://r-forge.r-project.org/projects/rinside/
- http://r-forge.r-project.org/projects/rcpp/

>
> > Do you think this discussion should be broadcast to one of your mailing
> > list ? which one ?
>
>
> Yes, in fact. I'm taking this to rkward-devel.
>
>
> Regards
> Thomas


-- 
Romain Francois
Independent R Consultant
+33(0) 6 28 91 30 30
http://romainfrancois.blog.free.fr