Akonadi: what should trigger fetching the full payload on new items resource-side?

[Daniel Vrátil]

On Friday, 24 September 2021 14:14:49 CEST Friedrich W. H. Kossebau wrote:
> Hi,

Hi Friedrich,

> 
> (cc: Daniel, having happily seen he might again have some spare resources
> for Akonadi? :) )

Let's hope it lasts :D

> 
> I found the following currently broken in Akonadi:
> 
> given a client monitoring items with a fetchscope with fullPayload, when a
> resource is bringing in a new collection with items in it and calling
> ResourceBase::synchronize() to make that known, currently nothing triggers
> the fetching of the payload into the server and thus the clients, so ETM
> e.g. filters them out.
> 
> Example:
> Add "Personal Contacts" with existing vcf files in ~/.local/share/contacts/.
> The contacts will not be displayed at current runtime, only after a restart
> of Akonadi and its clients, which seems to trigger some re-synchronization
> (not yet researched).

Although the case you are describing is valid, this example should work, since 
the vcard resource always sets a full item payload during item sync (see 
VCardResource::doRetrieveItem()/VCardDirResource::doRetrieveItem()), so if the 
Items are not showing up in clients, the problem is probably elsewhere. You 
might be tracking two bugs here.

I recommend manually checking the database to see if the payload is actually 
there or not.

> Now I wonder what logic in the current design should ensure and how that the
> payload is actually also pulled when a new item is created in the server
> and there are monitors subscribed which want the full payload.

If the Item doesn't have a full payload in Akonadi, it should be retrieved on 
demand whenever a client requests an operation that requires the full payload 
(usually an ItemFetchJob with fullPayload fetchscope, or ItemCopyJob and 
ItemMoveJob, in  some cases).

Historically, Akonadi::Monitor would internally ensure this, if the fetchScope 
of the Monitor was to fetch full payload and it received a notification about a 
new Item, the Monitor would schedule an ItemFetchJob with the same fetch scope 
and the server, upon finding that the client has requested more than what's 
stored in Akonadi, would fetch the payload form the resource and provide it to 
the Monitor. This way the Monitor was able to emit itemAdded() signal with 
full payload, even if the initial sync only synced items without payload.

The new Akonadi Server-side notification payload feature may have broken this, 
since the Monitor no longer uses ItemFetchJob - it should always received the 
entire Item already as part of the notification. The NotificationManager should 
take care of retrieving the missing payload, if there's at least one client 
who wants it. I would probably dig in here first, to see whether that's really 
the case.

Next step could be adding a validation to Monitor to check that the Item 
received as part of notification contains everything requested by the Monitor's 
fetch scope (which may not be as trivial as it sounds).

> I have spent some time already trying to understand the mechanisms (see also
> open MRs 71, 72, 76, 77 at
> https://invent.kde.org/pim/akonadi/-/merge_requests for things stumbled
> over during that), but any ideas how to fix it are currently impeded by a
> lack of full clue about the basic ideas behind the current Akonadi syncing
> designs.
> 
> E.g. should the ItemCreateHandler in the server already ensure that also the
> full payload is fetched during its own protocol, when there are
> subscriptions for that known to the NotificationManager?
> Or should it be something in the subscription notification server-side, only
> once the whole batch of new raw items is added?
> Or (unexpectedly) actually still something to be done by the client(s)?

This indeed should be taken care-of by the NotificationManager - it's the first 
place I'd go to check for a potential bug.

/Dan

> 
> Cheers
> Friedrich


-- 
Daniel Vrátil
www.dvratil.cz | dvratil at kde.org
IRC: dvratil on Freenode (#kde, #kontact, #akonadi, #fedora-kde)

GPG Key: 0x4D69557AECB13683
Fingerprint: 0ABD FA55 A4E6 BEA9 9A83 EA97 4D69 557A ECB1 3683
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.kde.org/pipermail/kde-pim/attachments/20210924/81e2fa66/attachment.sig>