D10702: Always use a job to delete files to avoid freezing process waiting on IO

[Mark Gaiser]

markg added a comment.


  In D10702#272901 <https://phabricator.kde.org/D10702#272901>, @meven wrote:
  
  > Here is the script I have been using : https://gist.github.com/meven/f0b2a36c61240e1d6e19753afd1d3d68
  >
  > My benchmark logic is :
  >  Create a folder with x files of k sizes.
  >  Copy this folder.
  >  Delete this folder using kfmclient, measuring the elapsed time
  >
  > I have used 100000 files of 1 byte, 100000 of 1kb, 1000 of 3mb, 3 of 1Gb.
  >
  > I ran this on a 2TB hard disc drive.
  >
  > Here are the very limited results:
  >
  >   In seconds
  >                   			avg
  >   before
  >   1b		1,586	1,738	1,684	1,66933333333333
  >   1k	        1,719	1,777	1,649	1,715
  >   3mb	        9,206	9,164	8,419	8,92966666666667
  >   1gb	        30,736	20,559	29,981	27,092
  >  
  >   after
  >   1b 		4,637	1,721	1,599	2,65233333333333
  >   1k	        32,186	1,726	1,685	11,8656666666667
  >   3mb	        2,491	7,287	7,896	5,89133333333333
  >   1gb	        1,464	13,344	17,271	10,693
  >  
  >
  >
  > It appears my benchmark methodology is mostly of no use due to huge outliners values.
  >  I am using kfmclient, but more than time I would need to measure the memory overhead also of using the kioslave instead of the fast-path but this could be tricky with cross-process execution.
  >
  > Also I think that if my patch was to get through we nay need to treat as a signature change: the behavior of the KIO::DeleteJob would change quite a bit from being most of the time synchronous to being asynchronous.
  >  Some App may have built on the assumption (knowingly or not) that the function only returns after the file(s) have been removed, which would not the case after this patch.
  >  An opt-in boolean option could be needed to trigger the new behavior while keeping the old one for applications that have been updated/reviewed yet and perhaps mark as deprecated the old behavior.
  >
  > Also I would like to take the time to add some tests, although I need to learn about how to write some and a .
  >
  > I would much apprieciate feedback.
  >  As the solution is not obvious and still in debatable to me, we could set up some IRC meeting. I will be hanging on #kde-fm.
  
  
  That downside is not a solution.
  Specifically with large folders, you'd really like to see some form of progression as it's being deleted. I would.
  
  Please take a good close look at my updated example of how to make this async:
  https://p.sc2.nl/ByvSb_-lQ
  I've added comments everywhere in there to explain how the code works and what needs to be done to use it in here.
  Just run it (you need C++14 and Qt) to see how it looks. (my qmake file incase you want it: https://p.sc2.nl/Hy25fO-xm)
  
  It keeps it on the client process (the performance optimization of David Faure) but offloads it to a different thread.
  I've extended the example to show the number of files that have been deleted every 300ms.
  That in it's own is slightly different to what the code currently does. Currently it calls slotReport after every 300 files. With this it would call slotReport after every 300ms. I don't think that's much of a problem.
  
  I can try and make a KIO patch for this if you want. As i now have the logic in that example code. (i just don't have a working frameworks dev environment at the moment)

REPOSITORY
  R241 KIO

REVISION DETAIL
  https://phabricator.kde.org/D10702

To: meven, #frameworks, dfaure, ngraham, #dolphin, jtamate
Cc: kde-frameworks-devel, jtamate, markg, ngraham, #frameworks, michaelh, bruns
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.kde.org/pipermail/kde-frameworks-devel/attachments/20180603/8fbf7c2b/attachment-0001.html>