[Kde-pim] Making offline/online handling sane

[Volker Krause]

Hi,

On Saturday 01 December 2012 18:48:59 Andras Mantia wrote:
>  I've been sturggling a lot whenever I travel with the fact that I cannot
> read my *offline* imap mail. Friday I run into it again and spent almost all
> my flight time to identify and fix the problem. What I discovered is not
> good.
> 
> The problem
> -----------
>   The visible problem is that if a resource is offline, and there is a
> request to it to deliver an item, you get an error:
> "Cannot fetch item in offline mode."
> 
> The cause
> ---------
>   For the above, the cause is pretty simple:
>    if ( !isOnline() ) {
>      const QString errorMsg = i18nc( "@info", "Cannot fetch item in offline
> mode." );
>      emit error( errorMsg );
>      return errorMsg;
>    }
> in ResourceBase::requestItemDeliveryV2.
> Sounds simple and reasonable at the first read, but here is a question: why
> do we have it in the ResourceBase?
> 
> The real problem
> ----------------
> 
> The answer to the above question is: because we don't know what "isOnline"
> means for a resource. There is a weird mix of its meaning in the Akonadi
> libraries and resources. Does it mean that the resource is turned off? Does
> it mean that the resource is not allowed to do network operations? Does it
> mean a mix of the two or something else?

It is supposed to mean "agent is operational". What exactly that means of 
course depends on the individual agent. Network availability is just one 
aspect, for resources connecting to a remote backend. File system not mounted 
(eg. for maildir), Nepomuk not being enabled, running on battery (eg. for the 
indexing agents), as well as persistent errors that require user actions (e.g. 
wrong password) are all conditions that influence this. And explicit user 
override of course.

So, the naming might not be optimal (isOperational?), and the implementation 
also has some issues with properly combining all those states, as you know 
from your recent fix in this area.

> Well, the problem is that right now it is a mix. From the above it means
> that Akonadi should not even *try* to get anything from a resoruce that is
> offline, as it will get an error, so it means "offline resource is turned
> off resource". But a resoruce is automatically turned offline if there is no
> network. That means a resource is offline when there is no network. And
> this is *generic*, the code is in AgentBase that turns them offline.

Right, but it's not unconditional. Agents need to explicitly enable this. So, 
local maildirs should not be affected by this.

> What the above means: no network, resource goes offline, if akonadi wants to
> get something from it, you get the error.
> 
> Even more: if a resource is offline, the task scheduler inside the resource
> does nothing. Task processing is shut down (see  void
> ResourceScheduler::scheduleNext() ). This means not even a local resource
> works on offline mode: no task are processed.

As mentioned above, they should not be affected by network state changes, not 
sure about the manual override in KMail though.

> The suggested solution
> ----------------------
> Hereby I suggest to bound the "online" state to the network operations.
> Online means a resource is *allowed* to perform network operations. Offline
> means a resource is *not allowed* to perform network operations. The Akonadi
> libraries (and here I mean AgentBase and ResourceBase) don't deal with
> online/offline state, the most they do is to signal the network state,
> although this would just replicate a Solid signal, so no real need for that
> either. Akonadi *should not* care about what the resources do or allowed to
> do.

This API wasn't added because Akonadi cares, but because every resource needs 
something like this. It's mostly convenience for resource authors.

> Online/Offline handling will be moved to the resources. If a resource
> doesn't work on the network, it just ignores it: it cannot be set offline.
> If it works on a network, if setOnline(false) is called, it must shut down
> all network connections and should not perform any network operation until
> setOnline(true) is called.
>  POP3 seems to do the right things, IMAP doesn't.

Don't they both just follow the network status?

> If it wants, it can give back items in another way (think of local cache per
> resource).

Local cache per resource makes no sense, that's what Akonadi is for, no?

> The questionmarks
> -----------------
> 
> I couldn't do a complete analysis of the whole code yet, I run out of flight
> time. :) And there are some questionmarks in my mind:
> - why can you read offline imap content if you set KMail to offline (but you
> cannot if you have no network): here I just have a guess: if the IMAP
> resource cannot connect to the server, it sets itself to offline mode
> (another confusion, as offline in this context might mean "broken"). If it
> is in that state, it starts to report errors back to the Akonadi server. My
> suspition is that "KMail in Offline mode", but with working connection
> doesn't set the imap resource completely offline and it performs some
> operations instead of reporting back errors (like cannot connect to the
> imap server).
> - why does Akonadi asks at all for items on a disconnected IMAP account,
> when everything should be in cache? Note that my wondering is only for the
> items. There is anyway the problem I said above, as IMAP sync folders on
> click, while it can't do it in offline mode. You end up with some sync task
> scheduled...


This I think is the real problem, and as we discussed on IRC, it's actually 
unrelated to the online/offline handling. Your (offline) IMAP resource for yet 
unknown reasons has a largely incomplete local cache. When Akonadi tries to 
answer KMail's requests for your mails it tries to fill the gap by asking the 
resource, which will fail if the resource can't connect.

You are not the first one with this issue, afiestas, Jesper and possibly David 
have similar issues in their IMAP caches. See http://www.afiestas.org/kdepim-
sprint-2-understanding-and-diagnosing-akonadi/ for diagnostics.

Once you are in there, there is no automatic and complete way to recover. This 
would need logic to download the missing parts during syncing. As a side-
effect this would fix switching from online to disconnected mode at runtime, 
as well as enable the "Spanish sync".

> - how many resources would be affected, who relied on the current offline
> interpration?

IIRC the filtering and indexing agents bind "online" to a different meaning. 
But considering the above, I'm not convinced we need drastic changes in the 
online/offline code at all, the causes for your problems are somewhere else I 
think.

> Anyway, I'd not mind if somebody would try to reproduce what I see to
> confirm it. How to do it?
> - set up a maildir and a disconnected IMAP account
> - sync them, especially the disconnected IMAP so you are sure mail content
> is downloaded
> - File>Work Offline in KMail: everything should work fine, you can switch
> between folder and read mail content
> - in akonadiconsole set the maildir resource offline and try to read mails
> (read those that you haven't read lately, so they are not in the cache)
> - unplug any network cable, shut down any wireless acount, if possible
> disable the wireless completely. Now try to read mail on the IMAP account
> and switch folders. Alternatively restart Akonadi/KMail after you disabled
> the network.
> 
> Tell me how it works.

This is pretty much my setup (local maildir + several disconnected IMAP 
accounts), offline use while traveling works perfectly fine. Maildir is always 
"online" (i.e it doesn't follow Solid), and only has a fraction of the data in 
the cache, if I force that to offline I of course can access my mails there 
anymore either. 

> If you agree with the above approach, I volunteer to come up a solution
> inside the akonadi libs (which will be easy, like removing code mangling
> offline and leave there only the interfaces to set a resource
> offline/online) and check the imap/pop3/mail dispatcher resources that they
> perform well and do what you should.
> 
> The above suggestion removes the feature of turning off a resource. In order
> to have that, we would need to introduce a new state for resources, but I'm
> not sure we need it at all.

I'm still not entirely sure I understand the proposed change, with "mangling 
offline" you mean combining (optionally) network status and user override in 
favor of just network? Seems like a step in the wrong direction to me. I'd 
rather make it more explicit why the resource if not operational, adding API 
for the scenarios mentioned above, instead of overloading the user override 
flag there. So that "offline" really means "not operational".

Regarding the need of a user override, didn't you insist on that for your POP3 
scenario that triggered your recent fix there? Also, until network detection 
is fully reliable (e.g. handling hotel wifi) we need it IMHO.

Ok, after reading the entire thread I think I understand the proposed change, 
but it's based on a misconception caused by the behavior of your not-really-
disconnected IMAP resource: The IMAP resource is not needed in order to access 
your email, it can be completely off. Therefore the proposed intermediate 
state of "don't use network but do some work nevertheless", while 
theoretically possible, is no needed by any resource right now I think. And 
even if so you can implement that by staying "online" (operational) despite 
the network being off (listening to Solid manually), and react to incoming 
tasks accordingly. None of this would fix your IMAP problem though.

regards,
Volker
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 190 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.kde.org/pipermail/kde-pim/attachments/20121208/94f8e9a2/attachment.sig>
-------------- next part --------------
_______________________________________________
KDE PIM mailing list kde-pim at kde.org
https://mail.kde.org/mailman/listinfo/kde-pim
KDE PIM home page at http://pim.kde.org/