[Nepomuk] Review Request: kinotify: Do not store the paths to be added

[Vishesh Handa]

> On Aug. 6, 2012, 12:17 p.m., Sebastian Trueg wrote:
> > Apart from the little superfluous return the patch is fine. I did not read it properly before. Who would have through that storing dir iterators instead of by arrays would make sure a difference?

It's not just iterators vs array.

Here is an example - Say I'm adding watches for a director "DirA" which has 50,000 subdirectories. The moment addWatchRecursively would enter DirA, it would store those 50,000 directory urls in the queue. Then it would process the queue one at a time, during the processing it would again keep adding directories to the queue. This results in a a large memory requirement.

In contrast, only one QDirIterator exists. It iterates over all the subdirectories one at a time, without saving any to a queue/list.

Do you see what I mean?


- Vishesh


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://git.reviewboard.kde.org/r/105892/#review16963
-----------------------------------------------------------


On Aug. 6, 2012, 10:03 a.m., Vishesh Handa wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> http://git.reviewboard.kde.org/r/105892/
> -----------------------------------------------------------
> 
> (Updated Aug. 6, 2012, 10:03 a.m.)
> 
> 
> Review request for Nepomuk and Sebastian Trueg.
> 
> 
> Description
> -------
> 
>     kinotify: Do not store the paths to be added
>     
>     kinotify used a QQueue<QByteArray> to store all the directories which
>     need to be added. Since we use QDirIterator, each path is presented as a
>     QString which is then encoded to its QByteArray (QFile::encodeName).
>     This results in *large* chunks of memory being allocated, and then
>     slowly being deallocated.
>     
>     Instead, we now use a QDirIterator, and do not store all the directories
>     which need to be added, we simply iterate over them. This way we do not
>     allocate large amounts of memory.
>     
>     There is a large performance improvement as well. On my system, with
>     38829 directories, adding all the watches now takes only about 10 seconds,
>     instead of about 65.
>     
>     Patch possible due to massif output provided by Jure Repinc <jlp at holodeck1.com>.
>     Thanks a lot.
>     
>     tldr: Use DFS instead of BFS -> Less memory consumption
>     
>     BUG: 304476
> 
> 
> This addresses bug 304476.
>     http://bugs.kde.org/show_bug.cgi?id=304476
> 
> 
> Diffs
> -----
> 
>   services/filewatch/kinotify.cpp e8843c8 
> 
> Diff: http://git.reviewboard.kde.org/r/105892/diff/
> 
> 
> Testing
> -------
> 
> kinotify tests pass successfully.
> 
> 
> Thanks,
> 
> Vishesh Handa
> 
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.kde.org/pipermail/nepomuk/attachments/20120806/07bba106/attachment.html>