EWS resource server-side notification handling problems

[Krzysztof Nowicki]

W dniu 2019-10-05 12:06:09 użytkownik Daniel Vratil <dvratil at kde.org> napisał:
> On Thursday, 3 October 2019 22:19:56 CEST Krzysztof Nowicki wrote:
>> When working with the EWS resource I'm experiencing some messy event flow
>> that leads to the folder getting out of sync in a fatal way causing the
>> need for a full resync. I'm seeking some advice how to fix this.
>> 
>> The problem happens in case of a modify+delete action (e.g. a message is
>> read and quickly deleted).
>> 
>> So here is what happens:
>> 
>>  1. The message attributes are modified (e.g. the message is read, causing
>> the SEEN attribute to be set).
>>  2. Akonadi calls itemsFlagsChanged() for the item, which triggers an
>>      UpdateItem EWS request.
>>  3. The message is deleted.
>>  4. Akonadi calls itemsRemoved() for the item, which triggers a DeleteItem
>> EWS request.
>> 
>> So far so good, but here is where the server-side notification subscription
>> kicks in:
>> 
>>  5. The EWS server sends a notification about the item having been updated
>> (in response to the request from step 2. The EWS server notification
>> mechanism does not care if the modification originated from the same
>> client. 6. The event triggers a collection sync for the folder, which
>> retrieves the list of changes on the server side and on the Akonadi side.
>>  7. The synchronization code sees that an item has been modified remotely
>> and looks for the matching local item in order to update it (at this point
>> I don't know that I have just modified the item). The lookup fails as the
>> item is already gone (step 3), at which point the code assumes that we're
>> seriously out-of-sync and triggers a full sync.
>> 
>> I have tried to somewhat mitigate this by logging items that were locally
>> modified or deleted and in case a notification is received for that
>> operation, it is ignored. This reduced the window of opportunity slightly,
>> but with the right timing it still happens.
>>
>> So I was wondering if the logic I use to handle server-side notifications is
>> correct. At one time I was thinking whether instead of triggering
>> collection syncs, I should let the notification handler apply the changes
>> manually by manipulating Akonadi items directly, but that could lead to
>> another amplification, as the resource would in turn get calls from Akonadi
>> about item operations that it has just made from the notification handler
>> thread.
> 
> It is safe to do a change to an Item from within the resource, as long as you 
> perform this on the default session. The Akonadi server is clever enough to 
> not send a change notification to the resource if the change was triggered by 
> the resource, since it assumes the resource knows what it did.
> 
> This only solves your race condition in part, though. Even when you would try 
> to apply the change reported by EWS from the resource back to Akonadi, you 
> would get an error if the item was deleted in the meantime. You would probably 
> need to try to fetch the item before trying to update it to see if it did not 
> disappear, but even that still leaves a small window for a race if the item is 
> deleted from Akonadi between the fetch and the modify jobs...I can't think of 
> a really air-tight solution to completely avoid races.

This is what makes me think more often to start migrating to using
MAPI instead of EWS, since in contrast to EWS, MAPI is session-aware,
so this problem does not occur. Both protocols could be used in
parallel (i.e. some requests could be done through MAPI, some through
EWS). This would also allow a nice gradual transition.

I know about previous attempts to use it, but nowadays it's much
easier, as there is no need to use DCE RPC any more, since everybody
moved to MAPI over HTTP, which is much simpler to set-up.

K.