[Kde-pim] Moving Trojitá to extragear

[Jan Kundrát]

On Friday, 23 November 2012 20:10:45 CEST, Kevin Krammer wrote:
> What I was mostly concerned with was if there is any IMAP 
> command that is not 
> in KIMAP but which you would need. I.e. there should be no 
> functional reason 
> not to use KIMAP when developing a Qt program that wants to talk to an IMAP 
> server.

I've skimmed through the (new) KIMAP code. This is my first time I'm walking through its internals, so I might be completely wrong. Pointers to where I'm wrong are welcome.

It looks like the IO communicaiton is done in a thread (sessionthread.cpp). That class more or less just shuffles bytes into strings or lists of data. All actual parsing seems to be delegated to individual jobs. Trojita does it in a different way, we have C++ classes for each supported IMAP response. All IO is done asynchronously and in the GUI thread (and we haven't had any issues with that, performance-wise). When a complete response is received, it gets queued and callers are free to act on the responses as they wish (note, a typical SELECT command triggers many, many responses, so you'll get many responses queued to act upon). In Trojita, all of that parsing is done by src/Imap/Parser/Parser.cpp and the related classes.

The bits where we start to differ significantly is when it comes to the reponse processing. It looks to me that KIMAP doesn't support command pipelining at all -- for example the SelectJob::handleResponse(const Message &response) in selectjob.cpp:142 appears to assume that it is the only job active at any given time. This has a real-world drawback, you cannot, say, select a mailbox and at the same time perform the LIST command. Trojita has no such limitation, responses are dispatched to a queue of all the running tasks, and they act in a FIFO-like manner. When a particular task (like the ListChildMailboxesTask) handles the LIST reponse, it sets a flag "ok, I've accepted this one" and that particular reponse is not delivered to the rest of the other active tasks (like the one reponsible for the SELECT command, etc). Yes, it's complicated, but it was very much worth the effort. See Trojita's src/Imap/Model/Model.cpp (function Model::responseReceived IIRC) for details of this re
 sponse routing if you're interested, it's also descriebd in my thesis about how Trojita works [1, p. 54-56].

It looks to me that the KIMAP library is a rather low-level one -- some IMAP commands have optional features (CONDSTORE, QRESYNC,...) that are missing. But the more important point is that simply having a tool for sending SELECT ... QRESYNC and parsing back VANISHED EARLIER responses is *not enough*; you have to maintain an extensive pile of stateful infromation if you want to achieve a reasonably good performance in IMAP. I don't see such code in KIMAP, it seems that it's in kdepim-runtime/resources/imap.

This is based on a list of classes which appear in the HTML docs after I run `doxygen Mainpage.dox` in kdepimlibs/kimap as of v4.9.2-346-g1c30d65 (yes, an ancient one, but that's what I have on disk).

> Is your low-level IMAP code tightly bound to that or is the 
> model simply using 
> your low-level/protocol code?

The part below the response dispatcher/router does not depend on the Model and upper layers at all; see src/Imap/Parser/ for what is in. This is the part which deals with converting between the raw byte stream and some nice C++ data structures.

> Those limitations lead to the creation of a new IMAP client library 
> implementation, KIMAP. While this does of course not change anything for 
> Trojita, the main concern is that it should make a difference 
> for anyone else 
> creating a new IMAP accessing program.
> Hence the question if there is anything missing in KIMAP that 
> you would need 
> would you start now.

KIMAP is missing some commands like THREAD, doesn't have anything for ESEARCH, doesn't support CONTEXT=*, cannot do COMPRESS=DEFLATE (this is not an exhaustive list). Error handling seems to be rather basic, with no detailed reporting on what was wrong with any given response.

But anyway, according to my experience, having the parser ready is less than 10% of the total work required to speak the IMAP protocol efficiently, so the more interesting bits are the layers which use the parsed responses and act on them. I haven't read through the Akonadi IMAP resource source code.

> My understanding of KMime is extremely limited, but as far as I can tell it 
> already shows its age, but on the other hand is well tested 
> through years of real-world usage.

I did not mean to imply otherwise. I was in a situation where I was looking for a RFC2047 decoder, not a MIME message parser. From that point of view, reimplementing RFC2047 yet again was easier than dealing with KMIME as a whole.

With kind regards,
Jan

[1] http://trojita.flaska.net/msc-thesis.pdf
_______________________________________________
KDE PIM mailing list kde-pim at kde.org
https://mail.kde.org/mailman/listinfo/kde-pim
KDE PIM home page at http://pim.kde.org/