Akonadi: what should trigger fetching the full payload on new items resource-side?

[Friedrich W. H. Kossebau]

Am Freitag, 24. September 2021, 17:25:47 CEST schrieb Daniel Vrátil:
> On Friday, 24 September 2021 14:14:49 CEST Friedrich W. H. Kossebau wrote:
> > (cc: Daniel, having happily seen he might again have some spare resources
> > for Akonadi? :) )
> 
> Let's hope it lasts :D

I do hope :)

((On that matter, next to the question of this email, I have another open task 
where your assistance would be good to have, even more as you started that 
task:
completing the introduction of the batch item retrieval, and getting rid of 
the current
    ResourceBase::retrieveItem(const Akonadi::Item &item, ..,)
API ideally in time for 21.12
I already adopted a few simple local filesystem-based resources to it, with 
one MR currently still open (kdepim-runtime MR 31). But the remote ones (IMAP, 
DAV, openXChange, tomboy) will need more clever people working on it, so 
perhaps you can find some interest in completing this open task as well? Happy 
to rather assist you here.))

> > I found the following currently broken in Akonadi:
> > 
> > given a client monitoring items with a fetchscope with fullPayload, when a
> > resource is bringing in a new collection with items in it and calling
> > ResourceBase::synchronize() to make that known, currently nothing triggers
> > the fetching of the payload into the server and thus the clients, so ETM
> > e.g. filters them out.
> > 
> > Example:
> > Add "Personal Contacts" with existing vcf files in
> > ~/.local/share/contacts/. The contacts will not be displayed at current
> > runtime, only after a restart of Akonadi and its clients, which seems to
> > trigger some re-synchronization (not yet researched).
> 
> Although the case you are describing is valid, this example should work,
> since the vcard resource always sets a full item payload during item sync
> (see VCardResource::doRetrieveItem()/VCardDirResource::doRetrieveItem()),
> so if the Items are not showing up in clients, the problem is probably
> elsewhere. You might be tracking two bugs here.

The "Personal Contacts" resource is that contactsresource though, which is 
something else than the vcarddirresource :) (also first had to sort that out).

The vcarddirresource is missing out some additional things on creation of a 
resource, which I though already have a plan for to handle, once this issue 
here is solved. So I stick to the contactsresource as example here.

And ContactsResource::retrieveItems(const Akonadi::Item::List &items, const 
QSet<QByteArray> &parts) is never called during the creation of a resource, as 
a result of nothing server-side seemingly being interested to satisfy the 
fullPayload flag set by the monitor. Things just end after 
ContactsResource::retrieveItems(const Akonadi::Collection &collection) has 
delivered all the new (payloadless) items to the server.

> > Now I wonder what logic in the current design should ensure and how that
> > the payload is actually also pulled when a new item is created in the
> > server and there are monitors subscribed which want the full payload.
> 
> If the Item doesn't have a full payload in Akonadi, it should be retrieved
> on demand whenever a client requests an operation that requires the full
> payload (usually an ItemFetchJob with fullPayload fetchscope, or
> ItemCopyJob and ItemMoveJob, in  some cases).
> 
> Historically, Akonadi::Monitor would internally ensure this, if the
> fetchScope of the Monitor was to fetch full payload and it received a
> notification about a new Item, the Monitor would schedule an ItemFetchJob
> with the same fetch scope and the server, upon finding that the client has
> requested more than what's stored in Akonadi, would fetch the payload form
> the resource and provide it to the Monitor. This way the Monitor was able
> to emit itemAdded() signal with full payload, even if the initial sync only
> synced items without payload.
> 
> The new Akonadi Server-side notification payload feature may have broken
> this, since the Monitor no longer uses ItemFetchJob - it should always
> received the entire Item already as part of the notification. The
> NotificationManager should take care of retrieving the missing payload, if
> there's at least one client who wants it. I would probably dig in here
> first, to see whether that's really the case.

one thing which caught my attention was that ItemCreateHandler::sendResponse() 
in some cases also would use ItemFetchHelper to ask during the protocol for 
the full payload. Though not in the case when a resource is running the 
ItemCreateJob as part of the retrieveItems(Collection) call.
When I first saw this, my initial thought was, okay, so the ItemCreateHandler 
should check the global notification fetch scope here and then ask the 
resource for all the missing payload data during the ItemCreate protocol.
Would that make sense, or is there chance of a deadlock? Still missing some 
parts of the bog picture, so unsure whether doing full payload query calls at 
this state would run into some issues at least with some resources, e.g. 
because there is that one-task-at-a-time thing with agents/resources I have 
yet to understand.

Cheers
Friedrich