[Kroupware] Re: [Kde-pim] Kaplan: The road ahead

[Steffen Hansen]

* Don Sanders <sanders@kde.org> [Sep 20. 2002 10:16]:
> On Thursday 19 September 2002 06:46, Zack Rusin wrote:
[...]
> > > For the actual synchronization algorithm we have two choices:
> > >
> > > 1) The dumb and simple one:
> > >     upload new messages in the cache to the server
> > >     delete deleted messaged in the cache on the server
> > >     download new messages from the server to the cache
> > >     delete deleted messages from the server in the cache
> > >
> > > 2) The complex one:
> > >     when offline { record all user commands on the cache in a
> > > command queue} when online {
> > >         play back recorded commands
> > >         delete deleted messages from the server in the cache
> > >         download new messages from the server to the cache
> > >     }
> > >
> > > I really hope we can get by with 1), as the other one is probably
> > > too timeconsuming to implement. Of course algo 1) potentially
> > > wastes some bandwidth compared to the other one, but I'd rather
> > > have something slow that works instead of something fast that is
> > > broken :)
> 
> Steffen method #1 requires keeping a list of deleted messages for each 
> IMAP folder right?

No, I think not. Every message is assigned a unique[1] and strictly
ascending number called the UID (this is done by the server). So
whenever we see a message on the server that has UID <= the last known
UID, and this message is missing from the cache, then it must have been
deleted locally.

Likewise, if we have a message in the cache with some UID <= last, and
this UID is no longer on the server, then it must have been deleted from
the server.

Same for new messages: New messages on the server have UID > last known
UID, and new messages in the local cache have not been assigned a UID
yet.

The UIDs are kept in the X-UID header in messages in the local cache,
and when a message is moved from a different folder, I make sure to
remove that header so the message appears new.

[1] Unique within the folder

Does this make sense?

> So that when the connection to the IMAP server is established the 
> messages in that list can be deleted from the IMAP server.
> 
> And won't you also need to keep a list of new messages, so that when 
> you connect to the IMAP server those messages will be uploaded to the 
> IMAP server?
> 
> I don't see how it's possible to know what has been deleted and what 
> is new by comparing what's in the local folder to what is on the IMAP 
> folder. Because someone might have logged into the IMAP server using 
> a second mail client and deleted messages.
> 
> In which case you need to keep lists of new and deleted messages, 
> right?
> 
> I'm just trying to understand the approach you are intending to take.
> 
> I think queueing commands is the right thing to do in the long term, 
> it would seem to be a pretty natural extension of the technique that 
> I just described. But keeping a queue of commands is going to require 
> merging commands in the queue so instead of
> delete message 1
> delete message 2
> ...
> delete message 100
> 
> A single IMAP kioslave delete job can created. This is desirable 
> because the IMAP kioslave is smart enough to issue a single range 
> deletion command when it can.

Yes, this is clearly on my todo list. Currently the code is a bit dumb
and just creates a job for each message that must be deleted. This will
change.

On the other hand, some jobs, like the ones fetching messages, might be
nice to have running "in parallel" to get the best of connections with
high latency and high bandwith.
 
> > Heh, yeah. But #2 wouldn't be that hard if I finally found some
> > time to port all folders to the new interfaces. Then we could have
> > a queue of KMFolderJobs for the specific cached folders.
> > KMFolderJob's don't start during construction like KMImapJob did,
> > but require the start() call to invoke them. So once we would have
> > the needed offline jobs constructed, we would wait till we connect
> > to the server, iterate through the queue once we are and call
> > start() on each of those jobs. This would also allow us to have a
> > nice graphical task-manager, which could be showing users what jobs
> > are pending.
> 
> I'm not sure that I follow you Zack.
> 
> As far as disconnected IMAP support goes I would think that in 
> KMFolderIMAP calling a method like delete or move, should immediately 
> act on the underlying folder (whether it be Maildir or mbox), and 
> also queue a KMFolderJob that would delete or move the message on the 
> IMAP server. Then the KMFolderJobs would stay queued until a 
> connection was available. So essentially offline and online would be 
> handled the same way.
> 
> The KMail GUI should just show the state of the Maildir or mbox folder 
> that is backing the IMAP folder.

Yes, that approach would have to apply the command to the local cache
right away in addition to queueing the command for the server.

wkr.
-- 
Steffen Hansen

Senior Software Engineer, Klarälvdalens Datakonsult AB

email: hansen@kde.org, steffen@klaralvdalens-datakonsult.se,
       steffen@hrhansen.dk
www:   http://www.hrhansen.dk