[Kde-pim] Moving Trojitá to extragear

Tue Nov 27 17:05:49 GMT 2012

On Friday 23 November 2012 21.55:39 Jan Kundrát wrote:
> On Friday, 23 November 2012 20:10:45 CEST, Kevin Krammer wrote:
> > What I was mostly concerned with was if there is any IMAP
> > command that is not
> > in KIMAP but which you would need. I.e. there should be no
> > functional reason
> > not to use KIMAP when developing a Qt program that wants to talk to an
> > IMAP
> > server.
> 
> I've skimmed through the (new) KIMAP code. This is my first time I'm walking
> through its internals, so I might be completely wrong. Pointers to where
> I'm wrong are welcome.
> 

Hey Jan,

Nice work with Trojita, it feels indeed very fast and I'm happy to see there 
is some development going on in this area =)
Note that I'm not an author of KIMAP but only a user, so I might not know much 
more about it than you do, I'm however interested in having a fast and 
reliable imap library in kdepimlibs, as not only does the imap resource rely 
on it, but I'm also using it for the Kolab Groupware server.

So IMO it would be great if we could find the common needs we have and make 
sure we have one IMAP implementation which suits us all.

> It looks like the IO communicaiton is done in a thread (sessionthread.cpp).
> That class more or less just shuffles bytes into strings or lists of data.
> All actual parsing seems to be delegated to individual jobs. Trojita does
> it in a different way, we have C++ classes for each supported IMAP
> response. All IO is done asynchronously and in the GUI thread (and we
> haven't had any issues with that, performance-wise). When a complete
> response is received, it gets queued and callers are free to act on the
> responses as they wish (note, a typical SELECT command triggers many, many
> responses, so you'll get many responses queued to act upon). In Trojita,
> all of that parsing is done by src/Imap/Parser/Parser.cpp and the related
> classes.
> 

If the additional thread is necessary or not I don't know, but it looks pretty 
much like an implementation detail to me that could be changed as we like (it 
seems to make sense if there is a lot of parsing to-do).

Your ImapTask looks very similar to our jobs, which are just objects to 
encapsulate async operations. The KJob base implementation does not support 
task dependencies, but something similar has been implemented in 
akonadi/job.cpp.

> The bits where we start to differ significantly is when it comes to the
> reponse processing. It looks to me that KIMAP doesn't support command
> pipelining at all -- for example the SelectJob::handleResponse(const
> Message &response) in selectjob.cpp:142 appears to assume that it is the
> only job active at any given time. This has a real-world drawback, you
> cannot, say, select a mailbox and at the same time perform the LIST
> command. Trojita has no such limitation, responses are dispatched to a
> queue of all the running tasks, and they act in a FIFO-like manner. When a
> particular task (like the ListChildMailboxesTask) handles the LIST reponse,
> it sets a flag "ok, I've accepted this one" and that particular reponse is
> not delivered to the rest of the other active tasks (like the one
> reponsible for the SELECT command, etc). Yes, it's complicated, but it was
> very much worth the effort. See Trojita's src/Imap/Model/Model.cpp
> (function Model::responseReceived IIRC) for details of this re sponse
> routing if you're interested, it's also descriebd in my thesis about how
> Trojita works [1, p. 54-56].
> 

For all I know KIMAP indeed doesn't support command pipe-lining, but it should 
IMO be possible to add support for that in session.cpp.
If would be a matter of matching jobs to responses, while making sure jobs are 
scheduled appropriately (i.e. the ones requiring a selected mailbox), but I 
think we could encapsulate that in the session pretty much transparently to 
the users of KIMAP (the session could just silently pipeline commands which 
are pipelinable from it's queue), and it would certainly be a very welcome 
addition for all imap clients.
In fact, akonadi/session.cpp implements a similar command pipelining (although 
it's currently disabled), as the protocol is modeled after IMAP, although 
there are no command dependencies AFAIK.

I've never actually implemented command pipelining, so tell me if I'm wrong, 
but it seems entirely feasible to me.

> It looks to me that the KIMAP library is a rather low-level one -- some IMAP
> commands have optional features (CONDSTORE, QRESYNC,...) that are missing.
> But the more important point is that simply having a tool for sending
> SELECT ... QRESYNC and parsing back VANISHED EARLIER responses is *not
> enough*; you have to maintain an extensive pile of stateful infromation if
> you want to achieve a reasonably good performance in IMAP. I don't see such
> code in KIMAP, it seems that it's in kdepim-runtime/resources/imap.
> 

Yes, KIMAP is all about sending and parsing commands, while that is certainly 
no everything required it's one of the essential building blocks and if we 
could shared that codebase I think everyone would profit.

Especially with you in-depth knowledge of IMAP I think we could improve our 
codebase in many ways and make it ready for the future. We'll need a fast and 
reliable imap implementation for the time coming =)

If that means we ditch KIMAP and use something entirely different or improve 
what's existing with what we need is open for me, but I hope we can avoid 
having multiple complete implementation of the same (I'm not proposing to 
ditch KIMAP just yet ;-).

Cheers,
Christian

> This is based on a list of classes which appear in the HTML docs after I run
> `doxygen Mainpage.dox` in kdepimlibs/kimap as of v4.9.2-346-g1c30d65 (yes,
> an ancient one, but that's what I have on disk).
> > Is your low-level IMAP code tightly bound to that or is the
> > model simply using
> > your low-level/protocol code?
> 
> The part below the response dispatcher/router does not depend on the Model
> and upper layers at all; see src/Imap/Parser/ for what is in. This is the
> part which deals with converting between the raw byte stream and some nice
> C++ data structures.
> > Those limitations lead to the creation of a new IMAP client library
> > implementation, KIMAP. While this does of course not change anything for
> > Trojita, the main concern is that it should make a difference
> > for anyone else
> > creating a new IMAP accessing program.
> > Hence the question if there is anything missing in KIMAP that
> > you would need
> > would you start now.
> 
> KIMAP is missing some commands like THREAD, doesn't have anything for
> ESEARCH, doesn't support CONTEXT=*, cannot do COMPRESS=DEFLATE (this is not
> an exhaustive list). Error handling seems to be rather basic, with no
> detailed reporting on what was wrong with any given response.
> 
> But anyway, according to my experience, having the parser ready is less than
> 10% of the total work required to speak the IMAP protocol efficiently, so
> the more interesting bits are the layers which use the parsed responses and
> act on them. I haven't read through the Akonadi IMAP resource source code.
> > My understanding of KMime is extremely limited, but as far as I can tell
> > it
> > already shows its age, but on the other hand is well tested
> > through years of real-world usage.
> 
> I did not mean to imply otherwise. I was in a situation where I was looking
> for a RFC2047 decoder, not a MIME message parser. From that point of view,
> reimplementing RFC2047 yet again was easier than dealing with KMIME as a
> whole.
> 
> With kind regards,
> Jan
> 
> [1] http://trojita.flaska.net/msc-thesis.pdf
> _______________________________________________
> KDE PIM mailing list kde-pim at kde.org
> https://mail.kde.org/mailman/listinfo/kde-pim
> KDE PIM home page at http://pim.kde.org/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.kde.org/pipermail/kde-pim/attachments/20121127/4f002e00/attachment.sig>
-------------- next part --------------
_______________________________________________
KDE PIM mailing list kde-pim at kde.org
https://mail.kde.org/mailman/listinfo/kde-pim
KDE PIM home page at http://pim.kde.org/