[FreeNX-kNX] Comments regarding the CONTRIB file

[Gian Filippo Pinzari]

Hi all,

please find my comments interleaded within:

=====

2. Code and Patches
-------------------

While FreeNX is a volounteer project, it of course depends on the work of 7
years kindly donated by NoMachine to the Free Software Community:

 >>> While I have been running some research on clustering, network computing
 >>> and possible improvements to the X Window system since 1997, actually I
 >>> started coding this in 1999, so I think it's 5 years now ;-).

2.1 Rootless nxagent

This is the most needed feature at the moment. It will allow using single
applications instead of a full featured desktop. Most code needed for that
is already in the nxagent source tree; its just still a bit buggy and needs
some "love".

Difficulty: Medium

 >>> The main problem with rootless nxagent is that it requires running the
 >>> Win32 X server in rootless mode as well. At the moment any new NX session
 >>> on Windows is run in its own X server. NX sessions (i.e. NX agents) should
 >>> rather run in a window of the X server, as it happens already on Linux/Unix.
 >>> This requires some work in nxclient (to change the way the Win32 X server
 >>> is managed/monitored) and in the Cygwin/NXWin server. The Cygwin/NXWin
 >>> server has not been extensively tested in rootless mode and may have bugs.
 >>> Even if these are minor problems for FreeNX (that mainly targets Unix),
 >>> they are important issues for NoMachine, whose user base is mainly formed
 >>> by Windows users.

2.2 Pseudo Color / True Color support

This is the second most important feature needed. With the addition of Pseudo
Color / True Color, it it possible to reconnect also on different depths (which
is not possible at the moment). This will also make nxagent independent of the
used Visuals (as far as I've understand it).

 >>> nxagent already supports TrueColor. I would rather say that it -only-
 >>> supports TrueColor. Supporting PseudoColor is not as important as it probably
 >>> appears. On the contrary, I don't think that PseudoColor is important at
 >>> all, except to support very old clients, designed for the monochrome displays
 >>> that were common in the eighties. I think you refer to something I mentioned
 >>> as "internal nxagent operation in 32 bits". nxagent should always operate in
 >>> 32bits mode internally and remap colors on-the-fly when accessing the real
 >>> display. This would make possible to adapt the output to 8/16/24/32 X servers
 >>> while keeping the same nxagent internals. It would also make possible to
 >>> reduce the number of colors of images, improving the compression results.

Last but not least, will this allow me to use Xvfb to keep a session running,
while no display is attached to it.

 >>> I think I get what you mean, but that's not the way this must be implemented.
 >>> At the moment nxagent doesn't need to be connected to a real display to
 >>> keep accepting protocol requests from the clients. The problem is that the
 >>> Xlib routines do not check if the display is valid (it's not their fault,
 >>> just the way they were designed), so to avoid fireworks nxagent should add an
 >>> "if" everytime it calls a X API and check that the display is still open.
 >>> This would also require better separation between the nxagent's X internals
 >>> and the X output it produces, that would be 100% independent from the real
 >>> X server that is connected at the remote side (as we have seen in the point
 >>> above). At the present moment it's easier and more efficient to just keep
 >>> the Xlib structures in place and hide the disconnection in Xlib/nxcomp.
 >>>
 >>> There are two alternatives:
 >>>
 >>> a. The proxy hides this, even to Xlib. This would be managed by the NX layer,
 >>>    as actually the agent is not connected to the X server but to the proxy.
 >>>    The agent would keep handling the X protocol but its output would be
 >>>    discarded by the proxy. The agent would obviously get a notification when
 >>>    the display is disconnected, so it could optimize some internals to skip
 >>>    the expensive operations, for example on images. The agent would receive
 >>>    a new notification at the time the display needs to be reconnected, so a
 >>>    new connection would be started by the proxy. Note that proxy is already
 >>>    able to persist to network failures and connect to new client proxies,
 >>>    after having optionally reset the channels. This design assumes that the
 >>>    proxy runs "close" to the agent, either in-process or on the same machine
 >>>    as the agent. Even if it is possible to handle the agent-proxy communica-
 >>>    tion between different machines, this would complicate things for the NX
 >>>    server, so much that it would make things unreliable.
 >>>
 >>> b. The details are managed inside Xlib. When the connection drops the proxy
 >>>    is terminated. A new proxy is instantiated upon reconnection. This is the
 >>>    current behaviour in NX 1.4.0.
 >>>
 >>> Implementing disconnected operation in Xlib (by modifying the transport layer
 >>> and the related functions in XlibInt.c) is not difficult. This is only a matter
 >>> of design. We have been evaluating to run the nxcomp compression stuff in-process
 >>> with the agent, mainly for performance reasons, so it would make a sense to go
 >>> for the solution at point a. The alternative is to run nxcomp in-process with
 >>> the nxssh process connecting to the agent, that could be slightly more ineffi-
 >>> cient from the X protocol point of view, but would permit to have multiple agents
 >>> (for printing, multimedia, etc.) share the same out-of-process proxy. Everything
 >>> revolves around how we want to handle access to local devices, but that's
 >>> another story ;-).

2.4 XDamage support

nxagent should be also able to profit from the new XDamage extension, to allow making
NX sessions again even faster.

 >>> XDamage is interesting for the potential it has of replacing VNC. Thanks to
 >>> this extension we could easily implement features like sharing the same user's
 >>> display among multiple NX sessions. This is very valuable in scenarios like
 >>> remote assistance and collaboration. It is unclear if this would also help NX
 >>> performances. I don't see how it could. XDamage is about "grabbing" the content
 >>> of the remote display in a more efficient way compared to XGetImage. NX never
 >>> grabs the remote display but it rather has the knowledge of what is on the
 >>> remote side and uses this knowledge to draw the display in the most efficient
 >>> way.

2.5 "Lazy image encoding"

Currently images are sent at once and just limited by having a small control channel
open to allow fast user interaction. It would be much better if the images would
be "streamed" in a way.

 >>> That's not correct. As stated in the documentation, images are already streamed
 >>> in the current version. The problem is that, while an image is streamed, the
 >>> client is put in a "suspended" state and it is restarted at the time the full
 >>> image has been recomposed at the remote side. This means that you get smooth
 >>> operation when running multiple clients (a client can't get all the bandwidth),
 >>> but if you open Mozilla and Mozilla tries to send a few megabytes of images,
 >>> it will become totally unresponsive. NX will not restart it until all the
 >>> images it needs to send are actually sent over the network. The reason is that
 >>> NX tracks the PutImage, not what the client is going to do with the target
 >>> Pixmap or Window once the PutImage has been transferred. Lazy encoding means
 >>> that the agent should track the Pixmap (and the Windows that are the target
 >>> of a CopyArea) and update them asynchronously, whenever bandwidth is available.

Quoting GFP:

"> What about doing this asynchronously? Not necessarily doing it immediately,
 > but analyzing (in parallel, the data is still sent with normal compression
 > regardless) what is seen and then biasing the type of compression over
 > time based on the "popularity" of given image characteristics for a
 > specific application or window class.

This is more or less what we want to do as part of the "lazy"
image encoding functionality that is going to be implemented.
The main goal of the "lazy" encoding is to decouple the image
handling from other protocol requests. The effect we want to
achieve is similar to loading a web page, where the browser
renders the images progressively, as they are downloaded from
the network. Once you have images sent asynchrounously, you
have opened the way to any form of post-processing."

[TODO: This part is not yet explained enough ]

 >>> This "asynchronous" handling of images was referred to the ability to select
 >>> the best encoding methods based on the characteristics of the bitmap (many
 >>> colors -> use JPEG, few colors -> use PNG/ZLIB). Once we have decoupled the
 >>> handling of PutImage requests from the content of the Pixmap/Windows that are
 >>> their targets, we can implement whatever batch-processing we like on the
 >>> images, including reducing the number of colors or sending them as B/W, an
 >>> operation that would be expensive to do in real-time.

2.6 Drag and Drop with automatic file transfer

The idea is:

Grab the X Drag And Drop requests in nxagent and advise the nxproxy on the other
side to stream you the file with the filename, which you have got. The file should
then be saved to a temporary location and the event given to the real application.
While the file is transferred nxagent should display some kind of status bar.

Difficulty: ~ Medium - Difficult

 >>> While I think you have probably understimated the difficulty of some of the
 >>> other implementations, this is actually quite easy to implement ;-). The problem
 >>> is that it doesn't make much sense. The X protocol already provides support for
 >>> transferring huge amounts of arbitrary data by means of Properties and Select-
 >>> ions. I think this is just a special case of "clipboard hanlding" and could
 >>> be implemented by letting two X client communicate by X. I'm not an expert of
 >>> such stuff, but probably the XDND specification can help here. Note also that
 >>> Property transfers can be "streamed" by splitting the payload in chunks. The
 >>> X protocol has already provision for that.

2.7 Add the GLX extension

Add the glx extension to nxagent and also try to remove roundtrips in GLX.

Difficulty: Easy - Medium

 >>> There are two ways to support the GLX extension:
 >>>
 >>> a. The easy way. Remap the GLX requests to the existing Screen functions in
 >>>    agent. This would translate most of the GL output in plain PutImages.
 >>>
 >>> b. The hard way. What we need is an "accelerated" implementation of the GLX
 >>>    extension in the nxagent X server. Actually nxagent should translate the
 >>>    GLX input in a GLX output, exactly the same way as we translate the RENDER
 >>>    protocol input in a RENDER output. This is possible, but requires implemen-
 >>>    ting, one-by-one, all the GLX Procs in the X server. This can be a few
 >>>    months of work.

===

Hope this helps to clarify some of the open issues.

Keep up with your great job ;-).

/Gian Filippo.

P.S.:  We hope to offer "bounties" on these and other implementations in the
next months. Let me know if there is anybody interested in partecipating.