KIMAP2

[Daniel Vrátil]

Hi Christian,

On Tuesday, November 15, 2016 3:13:09 PM CET Christian Mollekopf wrote:
> Hi,
> 
> I have largely completed the transition to KIMAP2 and will thus retire
> KIMAP. Since I know you are probably continuing to use KIMAP at least
> for some time, I made KIMAP2 completely coinstallable and namespaced
> everything into KIMAP2.
> However I will only be able to maintain KIMAP2, and not KIMAP. If you
> want to keep KIMAP around (I assume you do), I suggest we release it
> from a stable branch "kimap", as I would like to move master to kimap2.
> If that causes large problems with the way things are released let's
> discuss alternative solutions.

Since we are probably going to stick with KIMAP1 for some time in Akonadi/
KMail, I think we would still prefer to be able to follow the regular release 
workflow, i.e. stable branch, master devel, next stable branched from master 
etc. I think it would only be harder for release management to remember to do 
some special branching in case of KIMAP repository.

How about simply setting up kimap2.git repo with KIMAP2? Once/if we port 
Akonadi/KMail to KIMAP2, we can simply retire the KIMAP1 repository.

If there are interesting bugfixes in KIMAP2 that we would consider backporting 
to KIMAP1, we can always take care of it manually (we probably would no be 
able to merge anyway).

It also feels a bit more logical to me: having two releases of two products 
from a single repository would just lead to a lot of confusion...


> 
> In any case, I suggest to move to KIMAP2 in the long run.
> 
> So, here's what's changed with KIMAP2:
> 
> * KIMAP2 is now threading free:
> There no longer is a dedicated thread for each socket. The reasoning for
> this is manyfold:
>     * The threading strategy should be up to the application. The
>     current design imposes additional threads in any case.
>     * The threading resulted in lot's of subtle bugs and code-complexity
>     overhead for the synchronization.
>     * The whole design revolved around the idea that by running the
>     parser in a thread the parser can be blocking and the network will
>     always be the limiting factor. That assumption is not necessarily
>     true and the parser starts to behave very badly if we get data
>     faster than we can process it (The internal buffer will get large
>     and a majority of time will be spent on memcopying during the trim()
>     call).
>     * This design doesn't allow the consumer to regulate the amount of
>     data that enters the system. It reads as much into memory as it can,
>     which again assumes the network is the slowest part.
>     * During benchmarking of Sink the parser actually became the
>     bottleneck of the imap resource (because of the pathological
>     behaviour with memcopy in trim()).
> All of this directly leads to the next point.
> 
> * We have now a non-blocking parser:
>     * The parser operates with two buffers to emulate a ringbuffer =>
>     fixed memory usage (except for some dynamic allocation for large
>     literals).
>     * The parser never reads more data than it can process => If we
>     can't process fast enough the socket buffer will eventually fill up,
>     resulting in the server eventually stopping to send more data. Which
>     is how the network stack is supposed to work IMO.
>     * We open up possibilities for new streaming API to directly stream
>     i.e. attachments to disk.
>     * This resulted in the parser mostly vanishing from benchmarks and
>     memcpy to vanish entirely.

I did not look into the code, but is the parser decoupled from the socket 
publicly? I.e. can application move the socket into a secondary thread and run 
the parser in the main thread, just feeding it raw data from the socket? Or  
do I have to move the parser to secondary thread, and then have the jobs run 
in the main thread and receive data through queued connections?

> * We no longer depend on ki18n:
>     * This is a protocol implementation and not a place for translated
>     strings.

+1

> 
> * We no longer depend on kio:
>     * This was used for ktcpsocket only, and the only remaining benefit
>     of that was the KSslCertificateManager integration. This can easily
>     be readded externally by implementing the Session::sslErrors
>     callback and the Session::ignoreErrors function, as well as
>     QSslSOckets default ca certificate functions. (We may want to extend
>     the API there a bit).
>     * KIO has a whole slew of unrelated dependencies so this was an
>     important step to make KIMAP2 more selfcontained.

So is there some factory-like API that would allow us to provide our own 
KTcpSocket, falling back to QSslSocket by default? If not, would that be an 
acceptable change for you?

> 
> * I got rid of a lot of the result batching code:
>     * It complicated the code and was an optimization at the wrong level
>     IMO to save a couple of function calls.

+1

> 
> * Less overloaded result signals with lot's of parameters. One signal
> with a result struct containing all data.

+1

> 
> * The login job has received an overhaul.
>     * Removed the slightly confused EncryptionMode names that sometime
>     mixed the excryption and the use of starttls (which are two largely
>     unrelated things).
>     * Cleaned up the login logic which was a very complex statemachine
>     scattered accross different classes and is now only a complex
>     statemachine scattered over fewer classes =(
>     * Fixed a potential race-condition where we would send a CAPABILITY
>     request before receiving the greeting, which some servers seem to
>     handle anyways but some don't.
>     * Removed the encryption negotation which is handled by QSslSocket
>     according to the provided settings, and otherwise fails as it
>     should.

Cool :-) The login part is indeed complicated. I still have a SASL2 plugin 
that implements Google's XOAUTH authentication mechanism (basicall OAuth via 
IMAP), should probably try to upstream that to both KIMAPs :-)

> 
> There are some things left to do left and right, including some
> extensions that I'll likely implement (NOTIFY comes to mind), and some
> API cleanup,  but in general not much is standing in the way of a first
> release.
> I have not yet decided on a release date as so far as only Kube depends
> on it so far.

Again, this would be settled by having kimap2.git repo I think, then the 
different release needs of Kube and KMail would be solved...


Dan

> 
> You can find all code in the dev/kimap2 branch.
> 
> Cheers,
> Christian


-- 
Daniel Vrátil
www.dvratil.cz | dvratil at kde.org
IRC: dvratil on Freenode (#kde, #kontact, #akonadi, #fedora-kde)

GPG Key: 0x4D69557AECB13683
Fingerprint: 0ABD FA55 A4E6 BEA9 9A83 EA97 4D69 557A ECB1 3683
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.kde.org/pipermail/kde-pim/attachments/20161115/059eaa89/attachment.sig>