[Kde-pim] Review Request 117471: ItemSync: Use batch processing and allow throttling

[Christian Mollekopf]

> On April 15, 2014, 12:10 p.m., Dan Vrátil wrote:
> > akonadi/itemsync.cpp, line 471
> > <https://git.reviewboard.kde.org/r/117471/diff/1/?file=263950#file263950line471>
> >
> >     What if the last batch is smaller than batchSize? Won't this cause the syncer to wait indefinitely?
> 
> Christian Mollekopf wrote:
>     Yes, which I think is not avoidable, except with some arcane timeout, which I'd rather avoid.
> 
> Dan Vrátil wrote:
>     Well, the last batch will be smaller than batchSize in almost everytime (unless itemsCount % batchSize == 0), so this is quite serious problem IMO. It will leave the last couple items always unsynced. Or am I missing something here?

mDeliveryDone should be set to true if either mTotalItems == mTotalItemsProcessed or if deliveryDone was called. So I think this should work.


- Christian


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://git.reviewboard.kde.org/r/117471/#review55795
-----------------------------------------------------------


On April 10, 2014, 9:20 a.m., Christian Mollekopf wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://git.reviewboard.kde.org/r/117471/
> -----------------------------------------------------------
> 
> (Updated April 10, 2014, 9:20 a.m.)
> 
> 
> Review request for KDEPIM-Libraries.
> 
> 
> Repository: kdepimlibs
> 
> 
> Description
> -------
> 
> ItemSync: Use batch processing and allow throttling
> 
> With this patch we process the items in batches and gives resources the
> possiblity to throttle the retrieval of messages using the
> readyForNextBatch signal. This way we avoid using huge queries and
> loading all items into memory, making the whole process a lot less
> resource intensive, and make it work for large folder.
> 
> 
> Diffs
> -----
> 
>   akonadi/itemsync.h 32166d7154b0a9a5131b0dd215c0a5a75b2a67dd 
>   akonadi/itemsync.cpp 421c14a027836800bab1e62ca466d68475c71e1f 
>   akonadi/resourcebase.h 01f86a84c5fd78b8dfcabf638a881759b159f04c 
>   akonadi/resourcebase.cpp 4e5b1e9289d1cbfba42490961dab78ebaad8a586 
>   akonadi/tests/itemsynctest.cpp 4999c03abb25f3e1cf767b287dcf7698693745e9 
> 
> Diff: https://git.reviewboard.kde.org/r/117471/diff/
> 
> 
> Testing
> -------
> 
> I adapted the imap resource accordingly and I can now sync 200k mail folder with the bottleneck being mysql (as it should) and no significant cpu or memory usage by akonadi-server or the imap resource. Also, unittests. 
> 
> 
> Thanks,
> 
> Christian Mollekopf
> 
>

_______________________________________________
KDE PIM mailing list kde-pim at kde.org
https://mail.kde.org/mailman/listinfo/kde-pim
KDE PIM home page at http://pim.kde.org/