[Kde-pim] Trying to understand mail filtering handling with maildir resource
Martin Steigerwald
Martin at lichtvoll.de
Sun Jul 13 14:14:09 BST 2014
Hi!
In order to finally break^H^H^H^H^H fix up performance of maildir resource I try
to understand maildir filtering handling with maildir resource. Especially I
want to ask the question:
Why does filtering mails into a folder trigger a full folder sync including
comparing all files each time?
I started from libmaildir and the only case it compares full directory
contents seem to be in
838 void Maildir::refreshKeyCache()
839 {
840 KeyCache::self()->refreshKeys( d->path );
841 }
which calls:
26 void KeyCache::addKeys( const QString& dir )
27 {
28 if ( !mNewKeys.contains( dir ) ) {
29 mNewKeys.insert( dir, listNew( dir ) );
30 //kDebug() << "Added new keys for: " << dir;
31 }
32
33 if ( !mCurKeys.contains( dir ) ) {
34 mCurKeys.insert( dir, listCurrent( dir ) );
35 //kDebug() << "Added cur keys for: " << dir;
36 }
37 }
38
39 void KeyCache::refreshKeys( const QString& dir )
40 {
41 mNewKeys.remove( dir );
42 mCurKeys.remove( dir );
43 addKeys( dir );
44 }
which essential calls the routines I switched not to sort the filenames. The
routines which list all the filenames of a mail folder into a buffer.
Maildir resource calls this in:
711 void MaildirResource::slotDirChanged(const QString& dir)
712 {
713 QFileInfo fileInfo( dir );
714 if ( fileInfo.isFile() ) {
715 slotFileChanged( fileInfo );
716 return;
717 }
718
719 if ( dir == mSettings->path() ) {
720 synchronizeCollectionTree();
721 synchronizeCollection( Collection::root().id() );
722 return;
723 }
TODO: Figure out what these exactly do.
Documentation
void ResourceBase::synchronizeCollection ( qint64 id )
protected
This method is called whenever the collection with the given id shall be
synchronized.
Definition at line 1071 of file resourcebase.cpp.
isn´t exactly helpful. What does synchronize mean?
Hmmm, okay it calls an collection fetch job, so retrieves all items from it
724
725 if ( dir.endsWith( QLatin1String( ".directory" ) ) ) {
726 synchronizeCollectionTree(); //might be too much, but this is not a
common case anyway
727 return;
728 }
729
730 QDir d( dir );
731 if ( !d.cdUp() )
732 return;
733
734 Maildir md( d.path() );
735 if ( !md.isValid() )
736 return;
737
738 md.refreshKeyCache();
And this probably is in there to make sure KeyCache of libmaildir also has the
most recent view on things… but why? Shouldn´t it always have it.
So everytime the dir is changed this is called.
Yet I so not understand why it is called on filter. Thus I looked in filteragent
in kdepim… there is
445 bool FilterManager::processContextItem( ItemContext context )
446 {
[…]
459 if ( context.moveTargetCollection().isValid() &&
context.item().storageCollectionId() != context.moveTargetCollection().id() )
{
460 if ( itemCanDelete ) {
461 Akonadi::ItemMoveJob *moveJob = new Akonadi::ItemMoveJob(
context.item(), context.moveTargetCollection(), this );
462 connect( moveJob, SIGNAL(result(KJob*)),
SLOT(moveJobResult(KJob*)) );
463 } else {
464 return false;
465 }
466 }
which is called from
406 bool FilterManager::process( const Akonadi::Item& item, bool
needsFullPayload, const MailFilter* filter )
ItemMoveJob is in kdepim-runtime/filestore.
I looked there as well… but didn´t yet get, how these mail folder
synchronisations on filtering get called.
I am tempted to remove
738 md.refreshKeyCache();
for testing from the slotDirChanged method, but I do not really completely
understand the implications of it.
Is there a flow chart of mail filtering handling?
Anyone willing to explain to me how it works? Can make an appointment on IRC
and I will post the results here.
Any high level overview of how all this Akonadi stuff works, how the individual
parts interconnect with each other?
I also looked at the method reference earlier already, but it didn´t help me
to see the forest instead of the single trees.
I will probably try to dig deeper anyway. But for now this is where my
research is. All I assume is that a ton of needless work is involved. I would
expect mail filtering to just involve the
46 void KeyCache::addNewKey( const QString& dir, const QString& key )
47 {
48 mNewKeys[dir].insert( key );
49 // kDebug() << "Added new key for : " << dir << " key: " << key;
50 }
51
52 void KeyCache::addCurKey( const QString& dir, const QString& key )
53 {
54 mCurKeys[dir].insert( key );
55 // kDebug() << "Added cur key for : " << dir << " key:" << key;
56 }
57
58 void KeyCache::removeKey( const QString& dir, const QString& key )
59 {
60 //kDebug() << "Removed new and cur key for: " << dir << " key:" << key;
61 mNewKeys[dir].remove( key );
62 mCurKeys[dir].remove( key );
63 }
methods of maildir instead of hovering over all files in the folder over and
over and over again. These methods would just remove and add the one file of
the mail that is moved.
Making this change could superboost mail filtering with large maildirs big
time, I bet.
Also… as the name suggests this is a cache. So we have a cache in libmaildir
and a cache in Akonadi and a cache on the operating system level. Why?
If you have any hint that may help me to fit my single pieces of research
together to some grand enlightment on how Akonadi works, I am all ears. I am
holding a training next week again, I may be able to show up on IRC. Otherwise
its next weekend or after that.
I am willing to document things I learn.
Actually I think I am not missing all that much anymore. Just can´t connect
the single parts spread over several git repos together yet.
Ciao,
--
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599 84C7
_______________________________________________
KDE PIM mailing list kde-pim at kde.org
https://mail.kde.org/mailman/listinfo/kde-pim
KDE PIM home page at http://pim.kde.org/
More information about the kde-pim
mailing list