[FreeNX-kNX] NX server performance

[DRC]

In the ongoing quest to figure out an appropriate solution for integrating
VirtualGL with NX that achieves decent performance and usability, I've been
conducting an in-depth performance analysis of nxagent 3.1.0.  For those who
don't know the back story, VirtualGL redirects 3D rendering from an OpenGL
application to a dedicated rendering device and captures the 3D output as a
video stream.  From the point of view of nxagent, VirtualGL is basically
just like a video player, sending a continuous stream of large (1-2
Megapixel) XShmPutImage() requests.  But the performance has never been
acceptable with NX (generally 6-7 Megapixels/second, whereas with TurboVNC
we can get 25-30 Megapixels/sec.)  Even when we replace the JPEG codec in NX
with the same one used in TurboVNC (TurboJPEG, which is 3X faster than
libjpeg), the frame rate barely budges in NX.

My analysis shows that the frame rate in NX is primarily being limited by
nxagent, and furthermore, it seems to be limited because nxagent spends
literally half of its CPU time in two functions:  md5_process() and fbBlt().
The rest of nxagent's time is spent doing JPEG compression, which is what I
would expect.

So, a couple of questions:

-- Can anyone tell me whether the NX protocol does MD5 analysis on all data
that is sent to the client?  If so, is there a way to bypass this?  That
would explain the time spent in md5_process().

-- Does anyone know why fbBlt() would be called so often?  It appears that
about 80% of the calls to fbBlt are with the following stack:

fbShmPutImage -> ProcCopyArea -> damageCopyArea -> nxagentCopyArea ->
fbCopyArea -> fbDoCopy -> fbCopyRegion -> fbCopyNtoN -> fbBlt

whereas the other 20% are with various other stacks leading back up to
fbShmPutImage().  However, the odd thing is that it appears that fbBlt() is
called a total of 3 times on every image that is passed into
fbShmPutImage().  fbBlt() seems to be unchanged relative to the X.org code
base, but after examining the code, I'm still not sure what the function is
supposed to do or why it would be called multiple times.

Note that in all cases, I run NX with deferred updates disabled, Zlib
compression disabled, and encryption disabled.  So it should be basically
just sending the JPEG images over the wire in unadulterated form.

Darrell Commander
The VirtualGL Project
(http://www.virtualgl.org)