[Kde-pim] Speed of storing large numbers of records, and sizes of akonadiserver and dbus-daemon

[Shaheed Haque]

OK, I finally ditched the ItemSyncJob approach and reverted back to a
bare-metal approach, and that seems to have done the trick. It seems
fast, and the memory hogging has gone too. I've committed the new
design, and after a bit more testing etc., I hope to move onto the
resync/update part of the problem.

Thank you all for your help.

Shaheed

On 27 February 2012 11:48, Shaheed Haque <srhaque at theiet.org> wrote:
> Hi Andras,
>
> Fetching the pull payload might or might not be an issue (though it
> would certainly add overhead!), but I'm not sure what control I have
> over the set of items being fetched.
>
> IIUC, the records are fetched in order to determine which items are
> new, and which will need updates. It seems to me that the current
> logic is designed to handle the case where incremental updates are not
> in use, so the whole collection is fetched in order to know what items
> need updates/are new. In the incremental case, I would say that only
> the items passed in to setIncrementalSyncItems() are relevant to the
> check. Anyway, see retrieveGALBatch in:
>
> http://websvn.kde.org/trunk/playground/pim/akonadi/exchange/contacts/exgalresource.cpp?revision=1282168&view=markup
>
> The actual ItemSync job is created around line 844.
>
> Thanks for looking, Shaheed
>
> On 27 February 2012 07:04, Andras Mantia <amantia at kde.org> wrote:
>> Hi,
>>
>>
>> Shaheed Haque wrote:
>>
>>> I managed to catch the inital part of the debug output, and I *think*
>>> this is the command that is fetching all the results:
>>>
>>> akonadi_exgal_resource_269 (0x7fdb30030ec0) 35 FETCH 1:* FULLPAYLOAD
>>> ALLATTR CACHEONLY EXTERNALPAYLOAD (UID REMOTEID REMOTEREVISION
>>> COLLECTIONID FLAGS SIZE DATETIME)
>>>
>>> I assume the asterisk is the reason whay all the records are fetched?
>>> Assuming so, that seems to come from
>>> ItemFetchJobPrivate::startFetchJob(), which is started from
>>> ItemSync::doStart(). IIUC, the idea seems to be to fetch the entire
>>> collection, modify it, and then write the results using
>>> ItemModifyJob. The key line of code seems to be in
>>> ItemSync::doStart():
>>>
>>>   ItemFetchJob* job = new ItemFetchJob( d->mSyncCollection, this );
>>>
>>> Now, given that I am signalling my intention to modify only part of
>>> the collection by calling ItemSync::setIncrementalSyncItems(), how
>>> about if the above line was changed to moral equivalent of:
>>>
>>>   ItemFetchJob* job;
>>>   if ( d->mIncremental ) {
>>>        incremenatlItems = d->mRemoteItems + d->mRemovedRemoteItems;
>>>        job = new ItemFetchJob( incrementalItems, this );
>>>   } else {
>>>         job = new ItemFetchJob( d->mSyncCollection, this );
>>>   }
>>>
>>> AFAICS, that would be consistent with the definition of
>>> setIncrementalSyncItems(), and avoid fetching the whole collection.
>>> Any thoughts?
>>
>> Where is your code, can you give us a weblink or such? I assume somewhere in
>> playground, but just point us.
>> Maybe you didn't set the setItemSynchronizationFetchScope to not fetch the
>> full payload when something has changed?
>>
>> Andras
>> _______________________________________________
>> KDE PIM mailing list kde-pim at kde.org
>> https://mail.kde.org/mailman/listinfo/kde-pim
>> KDE PIM home page at http://pim.kde.org/
_______________________________________________
KDE PIM mailing list kde-pim at kde.org
https://mail.kde.org/mailman/listinfo/kde-pim
KDE PIM home page at http://pim.kde.org/