KDirWatch bug and the analysis. Help is welcome!

[Mark]

Hi,

I'm horrible in clearly explaining issues and i'm going to explain a
lot in this mail. Please read it very carefully.

= The symptoms =
When opening my massively huge test folder (500.000 0 byte files) for
a resorting test in dolphin (pressing the name Name column when in
details view) i noticed that the resorting was done within 2 seconds.
However, after that i still get a "Loading folder..." in the lower
right corner for another ~4 seconds during which the gui is blocked.
All i can do is cancel it.

The reason why that happens is that i have per folder view settings.
When i press on the Name in details view (or resort in any other way)
then those new settings will be stored in a hidden .directory file.
The updating of that .directory file causes the folder itself to get a
dirty signal from KDirWatch. Understandable because the folder content
did change. That dirty signal then causes KDirLister to re-scan the
entire folder.

Those are the symptoms of the issue.

= Symptom analysis =
Now that i know what's causing the "Loading folder..." seemingly
needless i can figure out why this happens. The base issue is that
KDirLister gets a dirty signal with the folder as argument. So for
example if my path to this massive folder is:
/home/mark/massive_folder (which it actually is)

Then the first dirty signal that arrives in KDirLister is:
(dirty) /home/mark/massive_folder -- causes full folder rescan

Where i would have expected:
(dirty) /home/mark/massive_folder/.directory

= Why is KDirWatch sending this false signal? =
That's the question that kept me bussy for the last couple hours. At
first i though that inotify was simply not capable enough to send
detailed signals of filed that aren't directly monitored, but only
monitored because the parent folder is monitored. Further
investigation on that part showed me that inotify is perfectly capable
of doing that so the issue wasn't inotify.

So then i started debugging KDirWatch to see what's going wrong.
Surprisingly enough much of KDirWatch is doing exactly what i was
expecting. If i print the signals that it gets from inotify then i end
up with only the following for a resort (which updates the .directory
file only)
(CREATE signal) /home/mark/massive_folder/.directory.lock
(MODIFY signal) /home/mark/massive_folder/.directory.lock
(CREATE signal) /home/mark/massive_folder/.directoryc13357.new
(MODIFY signal) /home/mark/massive_folder/.directoryc13357.new
(DELETE signal) /home/mark/massive_folder/.directoryc13357.new
(CREATE signal) /home/mark/massive_folder/.directory
(DELETE signal) /home/mark/massive_folder/.directory.lock

For all of the above, DELETE could also be MOVED. There are in the
same statement and i didn't dissect them further.

So based on all those signals, none is sending a plain:
/home/mark/massive_folder/

That lead me to think that KDirWatch itself was doing something that
it shouldn't do. Further investigation on that part brought me in the
direction of KDirWatchPrivate::slotRescan because that was being
triggered by the singleshot rescan_timer. That is where things go
fishy.
That function contains:
#ifdef HAVE_SYS_INOTIFY_H
    if (entry->isDir) {
      // Report and clear the the list of files that have changed in
this directory.
      // Remove duplicates by changing to set and back again:
      // we don't really care about preserving the order of the
      // original changes.
      QStringList pendingFileChanges = entry->m_pendingFileChanges;
      pendingFileChanges.removeDuplicates();
      Q_FOREACH(const QString &changedFilename, pendingFileChanges) {
        if (s_verboseDebug) {
          kDebug(7001) << "processing pending file change for" <<
changedFilename;
        }
        emitEvent(entry, Changed, changedFilename);
      }
      entry->m_pendingFileChanges.clear();
    }
#endif

The line:
emitEvent(entry, Changed, changedFilename);
is what's emitting a dirty signal (in KDirWatchPrivate::emitEvent) on
my home watched folder:
(dirty) /home/mark/massive_folder

Subsequent signals (even create, delete or anything besides dirty) is
following this same path where the emit is being triggered from
KDirWatchPrivate::slotRescan in the above mentioned snippet. All
signals are thus being send as dirty signals! For more clarity, this
is what KDirLister(Cache) actually receives (in
KDirListerCache::slotFileDirty) as signals for the same folder:
(DIRTY signal) /home/mark/massive_files
(DIRTY signal) /home/mark/massive_files/.directory.lock
(DIRTY signal) /home/mark/massive_files/.directorym13357.new
(DIRTY signal) /home/mark/massive_files (yes, it starts with this and
ends with this)

Now i'm guessing this isn't right. All dirty signals of which two(!!)
are my main folder.

= What about a fix? =
Well, i tried that. I think the rescan_timer should not be fired at
all in the inotify code paths. However, simply disabling it's shot
won't work because all current signals that come from inotify are
being emitted through this singleshot timer. Disabling it won't help
since then no signals will be send at all. However,
KDirWatchPrivate::inotifyEventReceived seem to have the right code
paths to send proper signals, all possible inotify emits paths have a
emitEvent(...), it's just not being called. The reason that isn't
being called is probably - big assumption here, i haven't figured that
part out completely - is because it only calls the emitEvent function
if it can find any clients for them. The event paths that come from
inotify:
/home/mark/massive_folder/.directoryc13357.new
/home/mark/massive_folder/.directory
/home/mark/massive_folder/.directory.lock

can't be linked to a client because those paths aren't registered.
Only the actual folder is:
/home/mark/massive_folder/

What i think might be the best solution is to just let the client
lookup be done for the actual path that comes from inotify:
/home/mark/massive_folder/.directoryc13357.new

And for it's parent:
/home/mark/massive_folder/

But i don't know if that's the best way. Will it break something? or
is there another way to fix this that might be better?


I hope i explained it all in a complete and sane manner and gave you a
full picture of this issue.
Any help/suggestions you folks can provide is certainly welcome!

Cheers,
Mark