[Kde-pim] Making offline/online handling sane

[Kevin Krammer]

Hi Andras,

On Wednesday, 2012-12-05, Andras Mantia wrote:
> Hi Kevin,
> 
> Kevin Krammer wrote:
> > Hi Andras,
> > 
> > thanks for looking into this,
> > 
> > On Saturday, 2012-12-01, Andras Mantia wrote:

> >> But a resoruce is automatically turned offline if there is
> >> no network. That means a resource is offline when there is no network.
> >> And this is *generic*, the code is in AgentBase that turns them offline.
> > 
> > Right, which again makes sense to ensure consistent behavior. We could
> > split it out in an easy to use helper class though, so agents that
> > currently call "needs network" could then instantiate the helper instead.
> > 
> >> Even more: if a resource is offline, the task scheduler inside the
> >> resource
> >> does nothing. Task processing is shut down (see  void
> >> ResourceScheduler::scheduleNext() ). This means not even a local
> >> resource works on offline mode: no task are processed.
> > 
> > Right. A resource or agent that is offline should not perform any work.
> 
> So far we agree how it works, which is good. :) What is not good, is that
> we don't agree this is good or bad. I think it is bad.
> The above feature is "turn off the resource", not make the resource not
> work on the network (what offline really would mean).

Offline means "not part of the system". Offline in not having network is one 
possibility.
I suggest we use different terms then. E.g. "TurnedOff" as for "Do not to 
anything", "StayDisconnected" for not attempting access of backend.

So in TurnedOff not doing anything is good, in StayDisconnected it might be bad 
(depending on what the agent can to without backend access).

> >> The suggested solution
> >> ----------------------
> >> Hereby I suggest to bound the "online" state to the network operations.
> >> Online means a resource is *allowed* to perform network operations.
> >> Offline means a resource is *not allowed* to perform network operations.
> >> The Akonadi libraries (and here I mean AgentBase and ResourceBase) don't
> >> deal with online/offline state, the most they do is to signal the
> >> network state, although this would just replicate a Solid signal, so no
> >> real need for that either. Akonadi *should not* care about what the
> >> resources do or allowed to do.
> > 
> > So if I understand this correctly online/offline would then only reflect
> > the state explicitly set by the user (manually or through an
> > application's policy).
> > Offlines as in "has been take offline".
> 
> Offline will mean that the resource is not allowed to do network
> operations. Yes, indeed this will be a request from the user or client
> application. If there is a network problem or some other problem, the
> resource should be in an error state (Online, Broken or such).

Ok. Assuming Offline in the meaning of StayDisconnected.

> >> Online/Offline handling will be moved to the resources. If a resource
> >> doesn't work on the network, it just ignores it: it cannot be set
> >> offline.
> > 
> > Doesn't that contradict what you've written above? If offline means the
> > user does not allow the resource to perform work, why would a user find
> > it OK if some resources ignore that wish?
> 
> I see no contradiction. Offline will not mean anymore the resource is not
> allowed to perform *any* work, it will mean it won't be allowed to perform
> network operations.

Again assuming Offline as StayDisconnected: isn't that independent of the 
backend access mechanism? E.g. if the backend is accessed via Unix socket?

> The reason this cannot be in ResourceBase or AgentBase is that we can't
> control what the resource does or not. We can either completely shut it
> down (like we do now) or not.

The resource handles that if it wants to (it can reimplement doSetOnline).
I think the problem is that this is not precise enough, since we have 
currently multiple concepts applied to a single boolean.

Or do you want to make sure every resource handles the state changes, as in 
making something pure virtual?

> What I propose is to make the offline a flag
> that the resource can use to turn the network operations off, if it needs
> network. If you still want the current offline=turned off behavior, we can
> keep it, but call it the right way.

Well, offline means both things so we need replacement for both meanings.
(if we reuse offline for one of them I expect confusion).

> What I want is to remove the confusion behind the offline term and of
> course to fix the IMAP resource, so I can read my mails also when I have
> no network.

That should work independent of the meaning associated with "offline".
This is what Akonadi server does.

> That's the whole point of the disconnected mode after all. ;)

Exactly!

> And I believe without making a distinction between offline and turned off
> there isn't a clear way to fix the problem.

Agreed.

> >> If it wants, it can give back items in another way (think of local cache
> >> per resource).
> > 
> > Akonadi *is* the local cache!
> > One of the major reasons to have it in the first place :)
> 
> Sure, I know. This doesn't mean it cannot have temporary caches. It was
> just an example. A local resource (maildir) might be a better example: it
> doesn't need network, so it shouldn't have an offline state.

Of course not offline as in "cannot create a TCP connection", but it could still 
not be able to reach its backend, e.g. removable volume removed.

Just like a user wanting the IMAP resource to cease network operations (e.g. 
expensive connection), the same user might not want a maildir resource to 
block a mount point.

> >> The questionmarks
> >> -----------------
> >> 
> >> I couldn't do a complete analysis of the whole code yet, I run out of
> >> flight time. :) And there are some questionmarks in my mind:
> >> - why can you read offline imap content if you set KMail to offline (but
> > 
> >> you cannot if you have no network):
> > Assuming "KMail offline" means KMail told all/some email resources that
> > the user wishes them to remain offline?
> 
> KMail Offline means that resources are manually set to offline mode. That
> is what KMail does. (File->Work Offline menu)

Reusing my suggested terms this would then switch them to StayDisconnected, 
right?

> >> here I just have a guess: if the IMAP
> >> resource cannot connect to the server, it sets itself to offline mode
> >> (another confusion, as offline in this context might mean "broken").
> > 
> > I don't think it means "broken" in the context of no network connection,
> > it rather means "backend not reachable due to external constraints".
> 
> We can introduce a different state for that, if needed.

Right. I think we will need it, user's don't want to get errors if the network 
is not available, they expect software to detect that.

> >> If it
> >> is in that state, it starts to report errors back to the Akonadi server.
> > 
> > If course, it is being asked for things it can't do right now.
> 
> Sorry, it is not clear for me what you mean here. Right now (with the
> current code), of course cannot report errors, as nothing reaches the
> resource in offline mode.

Hmm, misunderstanding then. I thought you were getting an error from code in 
resource base.

> >> - why does Akonadi asks at all for items on a disconnected IMAP account,
> >> when everything should be in cache? Note that my wondering is only for
> >> the items. There is anyway the problem I said above, as IMAP sync
> >> folders on click, while it can't do it in offline mode. You end up with
> >> some sync task scheduled...
> > 
> > No idea either, maybe some property of the cache policy incorrect or
> > missing?
> 
> This is something I need to investigate. A quick check at that time shown
> that "setCacheOnly(true)" was used, so it looked correct, but probably I
> missed something.

Right. This is currently your main problem. The IMAP resource is asked for 
something it should be asked for, in a state where it can't do what it is 
asked for. So the only thing it can do is report that as an error.
It should not have gotten the request at all.

> > As far as I can tell we have three different reasons why an agent is
> > offline: 1) user's explicit choice
> > 2) backend/service not available, e.g. backend is a remote server, no
> > network 3) agent has reached a "broken" state and doesn't want its
> > scheduler to dispatch new tasks until the problem is fixed.
> > 
> > Any other interpretation/reason come to mind?
> 
> I agree with the above, and I find it wrong.

Agreed, what I wanted to have is a list of things we currently map into that 
one boolean. We need to demulitplex them into individual states so each use 
case can change the appropriate one.

> >> Anyway, I'd not mind if somebody would try to reproduce what I see to
> >> confirm it. How to do it?
> >> - set up a maildir and a disconnected IMAP account
> >> - sync them, especially the disconnected IMAP so you are sure mail
> >> content is downloaded
> > 
> > What we expect after this stage is that all IMAP emails are fully cached
> > by Akonadi and all maildir mails have their envelopes cached, right?
> 
> Well, not really.

Are you sure?
After a complete sync with full payload caching I would expect mails to be 
served from cache (IMAP) and after a complete sync with envelope I would 
expect envelopes to be served from cache.

> The main problem is that even *cached* items are not
> reachable right now. With disconnected IMAP (when the full mail is cached),
> you cannot read the mails, because the IMAP resource is accessed in offline
> mode, but it cannot react because it is "turned off".

We agree on that is the problem, but this is not the expected behavior.
The expected behavior as described above is that the mails are delivered from 
cache, regardless of resource state. No?

> But what I would expect here is that offline means no network usage.
> Nothing else.

So: TurnedOff, NoBackendAcces and NoNetworkAccess?

> >> - in akonadiconsole set the maildir resource offline and try to read
> >> mails (read those that you haven't read lately, so they are not in the
> >> cache)
> > 
> > Expected result is that folders list their mails (all envelopes cached)
> > but mails cannot be displayed due to resource having been "switched off"
> > and bodies not having been cached?
> > But IMAP mails still working because their bodies have?
> 
> Indeed, the expected thing (even with the current implementation) is to be
> able to access everything that is cached.

Exactly.

> >> - unplug any network cable, shut down any wireless acount, if possible
> >> disable the wireless completely. Now try to read mail on the IMAP
> >> account and switch folders. Alternatively restart Akonadi/KMail after
> >> you disabled the network.
> > 
> > Since the resources are already offline by use intervention, we would
> > expect the situation to remain unchanged?
> 
> As I wrote somewhere above, turning the resources offline or unplugging the
> cable results in different behavior, even if visually in both cases the
> IMAP resource is just "offline".

Interesting. So it behaves correctly when switched off by the user (one of the 
previous steps) but starts acting differently when network is no longer 
available?

I would have expected that there is no difference to the previous step (when 
KMail has turned off the resources).

> >> If you agree with the above approach, I volunteer to come up a solution
> >> inside the akonadi libs (which will be easy, like removing code mangling
> >> offline and leave there only the interfaces to set a resource
> >> offline/online) and check the imap/pop3/mail dispatcher resources that
> >> they perform well and do what you should.
> >> 
> >> The above suggestion removes the feature of turning off a resource. In
> >> order to have that, we would need to introduce a new state for
> >> resources, but I'm not sure we need it at all.
> > 
> > I know it is an advanced feature but if possible I'd like advanced users
> > to be capable of telling the system not to download emails even if there
> > is a network connection.
> 
> Sure, that is what "offline" would mean (and what I fixed recently, when
> Akonadi silently turned the resources online, even if they were requested
> to stay offline). My suggestion would be a step forward: separate the "do
> not download my mails" case from "stop working for now completely".

Lets assume we have a distinct state "user does not want network operations".
If that is active, it will depend on what the resource's operations require. 

The IMAP resource would be in the same state as before if its server is not a 
local server, i.e. it cannot add/modify/remove items/collections nor retrieve 
items/collections.

A resource that only uses network for additional information, e.g. downloading 
avatar images, would continue to work normally.

Not a bad idea.

> > IIRC, during the migration work Laurent said that his use case is having
> > KMail running at home and at the office and switching the one he is
> > currently not using to "offline". He will probably not like it if he has
> > to remove that feature and quit the KMail instance he's currently not
> > using.
> 
> Sorry, I think you misunderstood me. I have the same thinking and problems
> like Laurent mentioned. I don't want to remove any feature like that. What
> I would remove *unless there is an opposition* is the way to completely
> turn off a resource.

Hmm. I guess if we have a state where we can tell resources not to access the 
backends, then most "offline" use cases are covered.
Still might make sense for agents, e.g. temporarily turning of the Nepomuk 
feeder agent.

> If you still want that, we can keep, but I will/want to rename the feature
> to "Turned off" instead of "Offline" and make "Offline" do what it should
> do: work, but don't touch the network.

As written above we'll need several more user switchable states.

Right now we have
- TurnedOff (if we want that)
- DoNotAccessBackend
- DoNotUseNetwork

Due to user expectations of software not attempting network access if there is 
not network available we might need a state that reflects the deteced 
unavailability of network, e.g. HasNoNetwork, and one that reflects detected 
unavailability of backend, e.g. HasNoBackend

detected as opposed to encountered during operation attempt. The latter would 
result in an error and "broken" resource state.

Cheers,
Kevin
-- 
Kevin Krammer, KDE developer, xdg-utils developer
KDE user support, developer mentoring
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 190 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.kde.org/pipermail/kde-pim/attachments/20121205/7b39585d/attachment.sig>
-------------- next part --------------
_______________________________________________
KDE PIM mailing list kde-pim at kde.org
https://mail.kde.org/mailman/listinfo/kde-pim
KDE PIM home page at http://pim.kde.org/