[Kde-pim] Trying to understand mail filtering handling with maildir resource

[Martin Steigerwald]

Am Sonntag, 13. Juli 2014, 15:14:09 schrieb Martin Steigerwald:
> Hi!
> 
> In order to finally break^H^H^H^H^H fix up performance of maildir resource I
> try  to understand maildir filtering handling with maildir resource.
> Especially I want to ask the question:
> 
> Why does filtering mails into a folder trigger a full folder sync including 
> comparing all files each time?
> 
> 
> I started from libmaildir and the only case it compares full directory 
> contents seem to be in 
> 
> 838 void
> Maildir::refreshKeyCache()                                                 
>                                                                     839 {
> 840   KeyCache::self()->refreshKeys( d->path );
> 841 }
> 
> which calls:
> 
>  26 void KeyCache::addKeys( const QString& dir )
>  27 {
>  28   if ( !mNewKeys.contains( dir ) ) {
>  29     mNewKeys.insert( dir, listNew( dir ) );
>  30     //kDebug() << "Added new keys for: " << dir;
>  31   }
>  32 
>  33   if ( !mCurKeys.contains( dir ) ) {
>  34     mCurKeys.insert( dir, listCurrent( dir ) );
>  35     //kDebug() << "Added cur keys for: " << dir;
>  36   }
>  37 }
>  38                                                                         
>                                                                            
> 39 void KeyCache::refreshKeys( const QString& dir )
>  40 {
>  41     mNewKeys.remove( dir );
>  42     mCurKeys.remove( dir );
>  43     addKeys( dir );
>  44 }
> 
> which essential calls the routines I switched not to sort the filenames.
> The  routines which list all the filenames of a mail folder into a buffer.
> 
> 
> Maildir resource calls this in:
> 
> 711 void MaildirResource::slotDirChanged(const QString& dir)
> 712 { 
> 713   QFileInfo fileInfo( dir );
> 714   if ( fileInfo.isFile() ) {
> 715     slotFileChanged( fileInfo );
> 716     return;
> 717   }
> 718 
> 719   if ( dir == mSettings->path() ) {
> 720     synchronizeCollectionTree();
> 721    synchronizeCollection( Collection::root().id() );
> 722     return;
> 723   }
> 
> TODO: Figure out what these exactly do.
> 
> Documentation
> 
> void ResourceBase::synchronizeCollection        (
>       qint64          id      )        protected
> 
> This method is called whenever the collection with the given id shall be 
> synchronized.
> 
> Definition at line 1071 of file resourcebase.cpp.
> 
> isn´t exactly helpful. What does synchronize mean?
> 
> Hmmm, okay it calls an collection fetch job, so retrieves all items from it
> 
> 724 
> 725   if ( dir.endsWith( QLatin1String( ".directory" ) ) ) {
> 726     synchronizeCollectionTree(); //might be too much, but this is not a 
> common case anyway
> 727     return;
> 728   }
> 729 
> 730   QDir d( dir );
> 731   if ( !d.cdUp() )
> 732     return;
> 733 
> 734   Maildir md( d.path() );
> 735   if ( !md.isValid() )
> 736     return;
> 737 
> 738   md.refreshKeyCache();
> 
> And this probably is in there to make sure KeyCache of libmaildir also has
> the  most recent view on things… but why? Shouldn´t it always have it.
> 
> 
> So everytime the dir is changed this is called.

Wait. Everytime a mail is moved to a different folder, in some sense that 
destination folder is changed…

… still I really want to believe that Akonadi differentiates between a change 
made within Akonadi like moving a mail due to a filter where it can exactly 
know which mail and which mail file is to be moved from which source folder to 
which destination folder and any change made outside of Akonadi and being 
catched by inotify watches.

In several places in maildir resource itself I see calls to a function to stop 
maildir scanning before a call into libmaildir, like in maildirresource.cpp:

419   stopMaildirScan( sourceDir );
420   stopMaildirScan( destDir );
421 
422   const QString newRid = sourceDir.moveEntryTo( item.remoteId(), destDir 
);
423 
424   mChangedFiles.insert( newRid );
425   mChangedCleanerTimer->start( CLEANER_TIMEOUT );
426 
427   restartMaildirScan( sourceDir );
428   restartMaildirScan( destDir );


This basically disabled the filesystem watcher:

855 void MaildirResource::stopMaildirScan(const Maildir &maildir)
856 {
857     const QString path = maildir.path();
858     mFsWatcher->stopDirScan( path + QLatin1Literal( "/new" ) );
859     mFsWatcher->stopDirScan( path + QLatin1Literal( "/cur" ) );
860 }
861 
862 void MaildirResource::restartMaildirScan(const Maildir &maildir)
863 {
864     const QString path = maildir.path();
865     mFsWatcher->restartDirScan( path + QLatin1Literal( "/new" ) );
866     mFsWatcher->restartDirScan( path + QLatin1Literal( "/cur" ) );
867 }  


Okay, it boils down to where

711 void MaildirResource::slotDirChanged(const QString& dir)

is called.

If its called on filesystem changes detected by inotify, I think thats fine.


But if its triggered on filtering a mail into a new folder, then I think thats 
a bug. Yet, moving a mail with maildirresource exactly should not trigger it.


Well, okay, got this far. It really seems to be called on filesystem changes:

maildirresource.cpp:139:  connect( mFsWatcher, SIGNAL(dirty(QString)), 
SLOT(slotDirChanged(QString)) );


yet, maildirresource stops scanning on moving a mail… so why it is 
synchronizing the whole folder? Need to closer check mailfilteragent and 
filestore then I bet.

Ciao,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7
_______________________________________________
KDE PIM mailing list kde-pim at kde.org
https://mail.kde.org/mailman/listinfo/kde-pim
KDE PIM home page at http://pim.kde.org/