Serializing possibly resource-intensive tasks such as meinproc

[Thiago Macieira]

Em Segunda-feira 11. Janeiro 2010, às 22.03.44, Alexander Neundorf escreveu:
> On Monday 11 January 2010, Lubos Lunak wrote:
> > somebody has already reported them various cmake's shortcomings in
> > dealing with parallel make when compared to unsermake.
> 
> Please do :-)

What is needed is for make (not cmake) to talk to the farm scheduler and know 
when it can start a new job. This way, we don't pay the fork()+execve()+setup 
penalty when the job is going to be sleeping for the next minute or two. 

My LD_PRELOAD solution allows me to avoid running 60 moc or 60 meinproc4 when 
I do make -j60, but I still run 60 tb that just sit there idling, waiting for 
the scheduler to tell it to go. Even with distributed jobs, my computer cannot 
preprocess fast enough, so from the 60 processes that start, only 10 or 12 
actually happen in parallel. The rest are just eating RAM and costing on 
startup.

I could enhance the LD_PRELOAD module to be this kind of local scheduler (or 
talk to it), stopping the process just before execve(). However, the fork() 
penalty is still there.

It also needs to split its jobs per category, since each category has a 
different kind of scheduling.
-- 
Thiago Macieira - thiago (AT) macieira.info - thiago (AT) kde.org
  Senior Product Manager - Nokia, Qt Development Frameworks
      PGP/GPG: 0x6EF45358; fingerprint:
      E067 918B B660 DBD1 105C  966C 33F5 F005 6EF4 5358
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 190 bytes
Desc: This is a digitally signed message part.
Url : http://mail.kde.org/pipermail/kde-buildsystem/attachments/20100112/06a2871f/attachment-0001.sig