D7995: KDevelop: address dirwatching inefficiency (WIP/PoC)

[René J.V. Bertin]

rjvbb added a comment.


  Then try it for yourself. For KDW to use QFSW as shown below, or use a project on a remote disk.
  
  And please read Milian's initial comments on the linked BKO ticket: the dirwatcher should not be watching individual files, as is evident from the signal handlers. Feeding those directories when you encounter them seems the evident way to go and as it happens it also addresses the event loop blocking that comes from letting KDW::addDir() do something similar.
  
  Some timings obtained earlier today, on my Linux rig:
  
  stock dirwatcher, deferred, no full project parsing
  
    %> time kdevelop5 -s 5:gcc ; time env KDIRWATCH_METHOD=QFSWatch kdevelop5 -s 5:gcc 
    kdevplatform.shell: Starting project import timer
    kdevplatform.shell: All projects imported in 18.509 seconds
    kdevplatform.filemanager: Project dir monitoring is enabled for project "gcc-7.2.0-MP"
    kdevplatform.shell:     creating dir watchers and/or starting parse jobs took an additional 0.984 seconds
    kdevplatform.filemanager: Deleting dir watcher took 0.025 seconds for project "gcc-7.2.0-MP"
    	28.147 user_cpu 4.985 kernel_cpu 0:33.94 total_time 97.5%CPU
    ...
    kdevplatform.shell: Starting project import timer
    kdevplatform.shell: All projects imported in 18.129 seconds
    kdevplatform.filemanager: Project dir monitoring is enabled for project "gcc-7.2.0-MP"
    kdevplatform.shell:     creating dir watchers and/or starting parse jobs took an additional 171.24 seconds
    kdevplatform.filemanager: Deleting dir watcher took 253.512 seconds for project "gcc-7.2.0-MP"
    	466.943 user_cpu 12.438 kernel_cpu 7:29.46 total_time 106.6%CPU
  
  
  
  with the current kdirwatch optimisation
  =======================================
  
    %> time kdevelop5 -s 5:gcc ; time env KDIRWATCH_METHOD=QFSWatch kdevelop5 -s 5:gcc
    kdevplatform.shell: Starting project import timer
    kdevplatform.shell: All projects imported in 20.846 seconds
    kdevplatform.shell:     starting parse jobs took an additional 0.002 seconds
    kdevplatform.filemanager: Deleting 4172 entry dir watcher took 0.036 seconds for project "gcc-7.2.0-MP"
    	31.532 user_cpu 5.100 kernel_cpu 0:35.83 total_time 102.2%CPU
    ...
    kdevplatform.shell: Starting project import timer
    kdevplatform.shell: All projects imported in 20.776 seconds
    kdevplatform.shell:     starting parse jobs took an additional 0.001 seconds
    kdevplatform.filemanager: Deleting 4172 entry dir watcher took 0.199 seconds for project "gcc-7.2.0-MP"
    	30.464 user_cpu 4.864 kernel_cpu 0:35.34 total_time 99.9%CPU
  
  Yes, doing a concurrent dirwatcher population increases the total import time from about 19.5s to a bit more than 20.5s. That increase gets lost in the noise for smaller projects. Of course that argument can be used against this patch, but only if you never use anything but a recent Linux kernel and local projects (and don't care about anyone else).
  
  You can argue all you want that KDW and/or QFSW should be optimised. Maybe they should, but the first step is always to be certain that you're using them as intended and not in a clearly suboptimal way. I won't claim this patch is the best way to achieve that but it is clearly better across the board than stock.

REPOSITORY
  R32 KDevelop

REVISION DETAIL
  https://phabricator.kde.org/D7995

To: rjvbb, #kdevelop
Cc: brauch, kdevelop-devel, geetamc, Pilzschaf, akshaydeo, surgenight, arrowdodger
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.kde.org/pipermail/kdevelop-devel/attachments/20170926/d63aceeb/attachment-0001.html>