[Kde-pim] Trying to understand mail filtering handling with maildir resource
Daniel Vrátil
dvratil at redhat.com
Mon Jul 14 10:13:23 BST 2014
On Sunday 13 of July 2014 20:07:17 Martin Steigerwald wrote:
> Am Sonntag, 13. Juli 2014, 15:14:09 schrieb Martin Steigerwald:
> > Hi!
> >
> > In order to finally break^H^H^H^H^H fix up performance of maildir resource
> > I try to understand maildir filtering handling with maildir resource.
> > Especially I want to ask the question:
> >
> > Why does filtering mails into a folder trigger a full folder sync
> > including
> > comparing all files each time?
> >
> >
> > I started from libmaildir and the only case it compares full directory
> > contents seem to be in
> >
> > 838 void
> > Maildir::refreshKeyCache()
> >
> > 839 {
> >
> > 840 KeyCache::self()->refreshKeys( d->path );
> > 841 }
> >
> > which calls:
> > 26 void KeyCache::addKeys( const QString& dir )
> > 27 {
> > 28 if ( !mNewKeys.contains( dir ) ) {
> > 29 mNewKeys.insert( dir, listNew( dir ) );
> > 30 //kDebug() << "Added new keys for: " << dir;
> > 31 }
> > 32
> > 33 if ( !mCurKeys.contains( dir ) ) {
> > 34 mCurKeys.insert( dir, listCurrent( dir ) );
> > 35 //kDebug() << "Added cur keys for: " << dir;
> > 36 }
> > 37 }
> > 38
> >
> > 39 void KeyCache::refreshKeys( const QString& dir )
> >
> > 40 {
> > 41 mNewKeys.remove( dir );
> > 42 mCurKeys.remove( dir );
> > 43 addKeys( dir );
> > 44 }
> >
> > which essential calls the routines I switched not to sort the filenames.
> > The routines which list all the filenames of a mail folder into a buffer.
> >
> >
> > Maildir resource calls this in:
> >
> > 711 void MaildirResource::slotDirChanged(const QString& dir)
> > 712 {
> > 713 QFileInfo fileInfo( dir );
> > 714 if ( fileInfo.isFile() ) {
> > 715 slotFileChanged( fileInfo );
> > 716 return;
> > 717 }
> > 718
> > 719 if ( dir == mSettings->path() ) {
> > 720 synchronizeCollectionTree();
> > 721 synchronizeCollection( Collection::root().id() );
> > 722 return;
> > 723 }
> >
> > TODO: Figure out what these exactly do.
> >
> > Documentation
> >
> > void ResourceBase::synchronizeCollection (
> >
> > qint64 id ) protected
> >
> > This method is called whenever the collection with the given id shall be
> > synchronized.
> >
> > Definition at line 1071 of file resourcebase.cpp.
> >
> > isn´t exactly helpful. What does synchronize mean?
> >
> > Hmmm, okay it calls an collection fetch job, so retrieves all items from
> > it
> >
> > 724
> > 725 if ( dir.endsWith( QLatin1String( ".directory" ) ) ) {
> > 726 synchronizeCollectionTree(); //might be too much, but this is not
> > a
> > common case anyway
> > 727 return;
> > 728 }
> > 729
> > 730 QDir d( dir );
> > 731 if ( !d.cdUp() )
> > 732 return;
> > 733
> > 734 Maildir md( d.path() );
> > 735 if ( !md.isValid() )
> > 736 return;
> > 737
> > 738 md.refreshKeyCache();
> >
> > And this probably is in there to make sure KeyCache of libmaildir also has
> > the most recent view on things… but why? Shouldn´t it always have it.
> >
> >
> > So everytime the dir is changed this is called.
>
> Wait. Everytime a mail is moved to a different folder, in some sense that
> destination folder is changed…
Yes, KDirWatcher emits a folder change for the folder when a new file is
created there. I remember I wrote a workaround for that somewhere, but can't
remember where...
> … still I really want to believe that Akonadi differentiates between a
> change made within Akonadi like moving a mail due to a filter where it can
> exactly know which mail and which mail file is to be moved from which
> source folder to which destination folder and any change made outside of
> Akonadi and being catched by inotify watches.
Nope, Akonadi receives a "MOVE" command and it does not really care who called
it, where from or where to the item(s) is/are being moved.
Akonadi does NOT know which file (the file in maildir) represents the email -
Akonadi has absolutely no knowledge of the end-storage (maildir folder). The
only storage Akonadi knows about is the database. Akonadi is generic storage -
the fact, that it does not know anything about the end storage and delegates
this to the Resources is the core concept of Akonadi.
> In several places in maildir resource itself I see calls to a function to
> stop maildir scanning before a call into libmaildir, like in
> maildirresource.cpp:
>
> 419 stopMaildirScan( sourceDir );
> 420 stopMaildirScan( destDir );
> 421
> 422 const QString newRid = sourceDir.moveEntryTo( item.remoteId(), destDir
> );
> 423
> 424 mChangedFiles.insert( newRid );
> 425 mChangedCleanerTimer->start( CLEANER_TIMEOUT );
> 426
> 427 restartMaildirScan( sourceDir );
> 428 restartMaildirScan( destDir );
>
>
> This basically disabled the filesystem watcher:
>
> 855 void MaildirResource::stopMaildirScan(const Maildir &maildir)
> 856 {
> 857 const QString path = maildir.path();
> 858 mFsWatcher->stopDirScan( path + QLatin1Literal( "/new" ) );
> 859 mFsWatcher->stopDirScan( path + QLatin1Literal( "/cur" ) );
> 860 }
> 861
> 862 void MaildirResource::restartMaildirScan(const Maildir &maildir)
> 863 {
> 864 const QString path = maildir.path();
> 865 mFsWatcher->restartDirScan( path + QLatin1Literal( "/new" ) );
> 866 mFsWatcher->restartDirScan( path + QLatin1Literal( "/cur" ) );
> 867 }
>
>
> Okay, it boils down to where
>
> 711 void MaildirResource::slotDirChanged(const QString& dir)
>
> is called.
>
> If its called on filesystem changes detected by inotify, I think thats fine.
>
>
> But if its triggered on filtering a mail into a new folder, then I think
> thats a bug. Yet, moving a mail with maildirresource exactly should not
> trigger it.
I think this ^^^ is the bug that you are hunting down - even though filesystem
watcher has been temporarily disabled for the move, the slotDirChanged is
still called - maybe it's called for parent folder? Or maybe
stopDirScan(path/{new,cur}) is not enough and some other folder has to be
unwatched temporarily?
Adding a simple qDebug() << dir to slotDirChanged() will tell you more.
>
> Well, okay, got this far. It really seems to be called on filesystem
> changes:
>
> maildirresource.cpp:139: connect( mFsWatcher, SIGNAL(dirty(QString)),
> SLOT(slotDirChanged(QString)) );
>
>
> yet, maildirresource stops scanning on moving a mail… so why it is
> synchronizing the whole folder? Need to closer check mailfilteragent and
> filestore then I bet.
I don't think that mailfilteragent plays any role in this. If what I wrote
above is true, then this is simply a bug in implementation of items move in
the Maildir resource, nothing more.
Dan
>
> Ciao,
--
Daniel Vrátil | dvratil at redhat.com | dvratil on #kde-devel, #kontact, #akonadi
KDE Desktop Team
Associate Software Engineer, Red Hat
GPG Key: 0xC59D614F6F4AE348
Fingerprint: 4EC1 86E3 C54E 0B39 5FDD B5FB C59D 614F 6F4A E348
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 490 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.kde.org/pipermail/kde-pim/attachments/20140714/19e1a7ad/attachment.sig>
-------------- next part --------------
_______________________________________________
KDE PIM mailing list kde-pim at kde.org
https://mail.kde.org/mailman/listinfo/kde-pim
KDE PIM home page at http://pim.kde.org/
More information about the kde-pim
mailing list