[Kde-pim] Review Request: Optimize file access in maildir

[Andras Mantia]

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://git.reviewboard.kde.org/r/103947/
-----------------------------------------------------------

(Updated Feb. 12, 2012, 1:35 p.m.)


Review request for KDEPIM and Kevin Krammer.


Changes
-------

Here is an updated version. Changes:
- the cache is in a common object for all maildirs (avoid copying them around, and problems if it is updated in one Maildir object, but not on the copy)
- keys not found in the cache are added automatically to it
- FIX: keys are removed from the cache if maildir removes the akonadi items before the files were deleted from the disk
- FIX: cache is forgotten and refreshed if the monitored directory changes outside of the resource. This is needed as we can't cleanly catch direct moves between the cur<->new folders done e.g by another app

With the extra deletes and refreshes, the code is slightly slower (~30-50ms in plus), still much faster then the original code


Description
-------

This patch reduces the number of file stats (QFile::exist) in the maildir library by caching the content of the cur/new directory listing and using QMap lookup to search for file existance.
The patch also optimizes the mapping between collections and Maildir objects in the resource.


Diffs (updated)
-----

  resources/maildir/libmaildir/CMakeLists.txt b8cfb92 
  resources/maildir/libmaildir/maildir.h b038f02 
  resources/maildir/libmaildir/maildir.cpp d61ac0b 
  resources/maildir/maildirresource.h 233732c 
  resources/maildir/maildirresource.cpp b5aa3e2 
  resources/maildir/retrieveitemsjob.h dfc7095 
  resources/maildir/retrieveitemsjob.cpp 97f22b3 

Diff: http://git.reviewboard.kde.org/r/103947/diff/


Testing
-------

1) make test in libmaildir passes
2) I run a variant of the patch for quite some time (~2 months) without problem. The only difference was using a QStringList instead of QMap for caching, which make the code even slower. :)
3) You can see the poor man's profiling with the QTime: with a 18K folder subsequent synching (when nothing changes, just click on the folder in KMail that triggers a sync), is around 1100ms. With the patch it is around 670ms. With a 30K folder it was above 2000ms, now it is below 1200ms. This is the time for "ENTRIESPROCESSED", the actual amount of time needed to compare the items in the database with the files on the disk.

Here are the full numbers (3 runs for each folder, numbers in ms):

18K folder, original code:
LOCALLISTDONE AT 1856 
ENTRIESPROCESSED AT 1101 

LOCALLISTDONE AT 1879 
ENTRIESPROCESSED AT 1098 

LOCALLISTDONE AT 1809 
ENTRIESPROCESSED AT 1097 


18K - with patch:
LOCALLISTDONE AT 1923 
ENTRIESPROCESSED AT 675 

LOCALLISTDONE AT 2002 
ENTRIESPROCESSED AT 674 

LOCALLISTDONE AT 2045 
ENTRIESPROCESSED AT 669 

Net gain is 300-400ms.

32K-original:
LOCALLISTDONE AT 3627 
ENTRIESPROCESSED AT 2041 

LOCALLISTDONE AT 3826 
ENTRIESPROCESSED AT 2009 

LOCALLISTDONE AT 3698 
ENTRIESPROCESSED AT 2024 


32K -with patch:
LOCALLISTDONE AT 3447 
ENTRIESPROCESSED AT 1304 

LOCALLISTDONE AT 3728 
ENTRIESPROCESSED AT 1216 


LOCALLISTDONE AT 3723 
ENTRIESPROCESSED AT 1207 

Net gain is 900-1000ms.


Thanks,

Andras Mantia

_______________________________________________
KDE PIM mailing list kde-pim at kde.org
https://mail.kde.org/mailman/listinfo/kde-pim
KDE PIM home page at http://pim.kde.org/