[FreeNX-kNX] Memory leak in nxcompext Png compression, patch included

[Mario Becroft]

I found the memory leak mentioned by me in
<87prgnm2s0.fsf at server.ak.quickcircuit.co.nz> on the 12th of March. The
symptoms are that nxagent (or potentially a client using the nx
libraries) gradually grows in size, for example to greater than 1
gigabyte, and gradually becomes slower. This is particularly noticable
when running certain programs that use many images.

It is caused by a bug in libnxcompext which leaks one 768-byte
allocation each time an image is created having <= 256 unique colours
and with Png compression enabled.

The bug is in Pgn.c:PngCompressData(). When an image has <= 256 unique
colours, a colour palette is allocated with png_malloc(). png_free() is
called to free the palette memory after png_destroy_write_struct(). This
is invalid, since png_destroy_write_struct() invalidates the png
structure and sets png_ptr to NULL. png_free() tests for a NULL png_ptr
and fails gracefully (there is no crash) but the memory is not freed.

The fix is to move the call to png_free() above the call to
png_destroy_write_struct() as per the following patch.

Anyone using NX with Png compression needs to apply this patch, and it
should be merged to mainline.

png_palette_free.patch: Fix png palette memory leak in libxcompext

--8<---------------cut here---------------start------------->8---
--- nxcompext/Pgn.c.1	2009-03-14 22:00:09.978820803 +1300
+++ nxcompext/Pgn.c	2009-03-14 21:50:16.624657977 +1300
@@ -548,13 +548,14 @@
   }
 
   png_write_end(png_ptr, NULL);
-  png_destroy_write_struct(&png_ptr, &info_ptr);
 
   if (color_type == PNG_COLOR_TYPE_PALETTE)
   {
     png_free(png_ptr, palette);
   }
 
+  png_destroy_write_struct(&png_ptr, &info_ptr);
+
   /*
    * Check the size of the resulting data.
    */
--8<---------------cut here---------------end--------------->8---

-- 
Mario Becroft <mb at gem.win.co.nz>