question and method about taglib performance

[Scott Wheeler]

On Sep 19, 2012, at 11:05 AM, "Yao Weijian (CM/ESW13-CN)" <Weijian.Yao at cn.bosch.com> wrote:

>      We checked the issue and found that much time was wasted when file fopen and fread. We think this is caused by fileIO. So we used mmap instread of fopen/fread/fwrite when OS is linux, and got the result that the performance was increased about 25%.

I'm pretty skeptical of that.  Are you sure that it wasn't just being moved into either kernel execution or that the speedup was from warm disk buffers?

That said, if you've got a patch, it's easy enough for us to take a look.  5000 files isn't an especially large collection though.

>       But, as we all know, when using file mmap, the  memory which the process need is equal to the size of file. For example, when the filesize is 1G, the process need 1G memory for file mmap. This is intolerable. So we add a condition. When filesize is smaller than 10M, we use mmap, others we use fopen which the taglib use now.

That's not actually true.  It's needs that amount of *address space*, not physical memory.  You can map files with mmap into memory that far exceed the physical memory.  mmap just, well, maps, it doesn't copy the file into memory.

The actual limit that would be of concern would be the 4 GB limit on address space in 32-bit processes.  In 64-bit processes you can map files as large as desired.

-Scott