[Kroupware] Kolab iCal <-> IMAP converter

[Diego Rivera]

--------BIG NOTE!-------
> | IMAP is the ultimate |
> | authority for the    |
> | ical events!!        |
> ------------------------
> Any time a change happens in the IMAP server, the ical file needs to be
> re-exported to the Web Server so that clients pulling the ical file
> don't force an ical generation on every request.  This includes
> uploading an ical file to the Web Server interface.  Can the IMAP server
> notify us of changes or run external programs on changes.  We really
> don't want to have to add this feature to Cyrus IMAP.
> 
> imap2ical
> ---------
> 0. Authentication?
> 1. Get list of e-mails in Calendar folder
> 2. Create array of ical events from e-mails
> 3. Create master ical file from list

I seem to remember something called jical which does something like
this, but it's J2EE (runs on JBoss), and it hasn't seen much updating
for a while.  We could use that for ideas though...

To mitigate the impact of ical generation on every request, a simple
caching approach might help:

i.e.: the user hits http://www.server.com/getical.php?user=someuser with
Evolution, a browser, Outlook, etc.

   The php page checks if the ical stuff is up-to-date with respect to
the directory (checking normal modification times using imap_check()). 
If the file for "someuser" doesn't exist or is out of date, it generates
it.  It then returns the contents of the generated file (whether it was
updated or not).

This isn't overly sophisticated, but it should do the trick.  file-based
mutex locking could be used to avoid duplicate generation on concurrent
requests.  The caching could even be improved upon to examine only those
messages that have changed, etc.

This approach generates the file greedily, and shouldn't be much of a
load increase.

I don't particularly like polling cuz it can get out of hand on large
systems, but if even basic caching is implemented I think it would
lessen the load considerably.

PLUSSES:
=============

    - IMAP remains the ultimate authority for ical events, since IMAP is
being used to retrieve them for generation anyway.

    - on overly active systems where users are constantly changing their
ical stuff, their free/busy isn't updated until somebody needs to get it
via http - thus a potentially lesser load.  Even less if few users have
authorized the proxy user to read their ical info and publish it.

    - if the info is up-to-date, it just gets cached and re-used
everytime until there's a change.

    - if the info changes constantly for every user, this still *at
worst* causes essentially the same load as the non-polling approach that
would generate the ical every time a change is made.

    - the "publish proxy" user can selectively be authorized ot publish
info for particular real users by using ACL's for read access.  One
doubt I do have is if these ACL's as supported by Cyrus can also be used
to allow/deny access to individual messages in a mailbox.  This could
allow a user to use a "publish these, but not those" approach when
publishing their FB stuff.

    - can be moved to a separate server box transparently to help reduce
the load.

    - PHP accelerators could be put in place.  Better yet -
faster-executing tools could be used to write the CGI, even a native
executable.

    - Handling of deleted events is transparent, since only events that
exist in the "calendar" folder get published.  I assume those are valid
events, and deleted events get put elsewhere...i.e., a "Deleted
Calendar" folder?  At the very least, they would have some qualifying
characteristic to set them apart (and could thus be safely ignored).

MINUSES:
=============

    - the polling approach is well-known to have the potential of
swallowing a system if not implemented correctly

    - this is only as effective as the resolution of the timer checks
done in php_imap::imap_check() et al (1 second I believe) - i.e., an (i
believe) unsolvable race condition could cause the file to not be
generated accurately if the following events occurr within 1 second:

    a) user X changes calendar info
    b) user Y requests the info via the script
        this causes regeneration to occur because of a)
    c) user X changes calendar info again
    d) user Z requests the info, and since the mailbox mod time
       is potentially  "the same" as the existing file, no
       regeneration occurs and the event from c) is "lost" until 
       the next update requested at least a second after b)
       occurred.

Granted, proper checking of the imap folder's information (i.e, change
date, number of new messages, etc) can allow a fairly smart script to
detect this and avoid it.  Also, caching this information for subsequent
comparison might prove to be the way to go (although it adds load to the
operation).

The only other way I see for this (short of writing the stuff into
Cyrus-IMAP) is "tight-loop polling" using select() or something like
that, but that is a potential cpu hog - especially with large numbers of
users.

Opinions?

Should I go and get my dunce cap again?  :)

> ical2imap
> ---------
> 0. Authentication
> 1. Split ical file into events
> 2. Get ical events from mail server
> 3. Process incoming events against the Deleted Events Log
> 4. Update events in mail server
> 5. Return modified ical file

I'm dying to see who's going to step up to the plate on this one, since
I don't have much of a clue on how to do this very cleanly while keeping
with the current architecture/design goals and philosophy!!!! :)


Best

-- 
===========================================================
* Diego Rivera                                            *
*                                                         *
* "The Disease: Windows, the cure: Linux"                 *
*                                                         *
* E-mail: lrivera<AT>racsa<DOT>co<DOT>cr                  *
* Replace: <AT>='@', <DOT>='.'                            *
*                                                         *
* GPG: BE59 5469 C696 C80D FF5C  5926 0B36 F8FF DA98 62AD *
* GPG Public Key avaliable at: http://pgp.mit.edu         *
===========================================================
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url : http://mail.kde.org/pipermail/kroupware/attachments/20030527/3a2cbf9f/attachment.bin