KIMAP2

[Christian Mollekopf]

On Wed, Nov 16, 2016, at 12:22 AM, Daniel Vrátil wrote:
> On Tuesday, November 15, 2016 11:51:54 PM CET Christian Mollekopf wrote:
> > On Tue, Nov 15, 2016, at 09:56 PM, Daniel Vrátil wrote:
> > > On Tuesday, November 15, 2016 3:13:09 PM CET Christian Mollekopf wrote:
...
> > > > * We have now a non-blocking parser:
> > > >     * The parser operates with two buffers to emulate a ringbuffer =>
> > > >     fixed memory usage (except for some dynamic allocation for large
> > > >     literals).
> > > >     * The parser never reads more data than it can process => If we
> > > >     can't process fast enough the socket buffer will eventually fill up,
> > > >     resulting in the server eventually stopping to send more data. Which
> > > >     is how the network stack is supposed to work IMO.
> > > >     * We open up possibilities for new streaming API to directly stream
> > > >     i.e. attachments to disk.
> > > >     * This resulted in the parser mostly vanishing from benchmarks and
> > > >     memcpy to vanish entirely.
> > > 
> > > I did not look into the code, but is the parser decoupled from the socket
> > > publicly? I.e. can application move the socket into a secondary thread
> > > and run the parser in the main thread, just feeding it raw data from the
> > > socket? Or  do I have to move the parser to secondary thread, and then
> > > have the jobs run in the main thread and receive data through queued
> > > connections?
> > The parser is not decoupled from the socket publicly. The session
> > (+loginjob) handles
> > the connection establishment + reaction to connection teardown and is
> > thus coupled to the
> > socket implementation.
> > 
> > Running anything in a separate thread thus means having the session
> > (that wraps the socket),
> > in that thread and running the job in the same thread.
> > 
> > IMO if performance or non-blocking behavior are the aim you should do as
> > much work as possible in the thread,
> > and not just parsing, so I don't see the point of a thread running just
> > the parser.
> > Maybe I just don't understand what you have in mind though, what would
> > be the usecase?
> 
> The usecase is to run the socket in the thread, that's all I care about. 
> Having the parser there too is an added bonus. Network can be blocking -
> even 
> if you use the "async" API of QSslSocket, certain operations that call 
> directly into openssl can still block (especially connectToHost), so I
> want to 
> avoid having that in the main thread of the resources, because then DBus
> calls can timeout and Akonadi has to wait for replies longer than necessary.
> 

I see. I noticed connectToHost getting hung up on some host lookup
routine
for insane amounts of time too and was wondering what that was about.
Can we narrow down which calls exactly are problematic?
I don't assume reading from the socket if we got a signal that new data
is available would
indeed block for any significant amount of time?
If it's i.e. only connectToHost() a simple solution may very well be to
simply wrap that call in
QtConcurrent::run with minimal complexity overhead.

> > 
> > > > * We no longer depend on kio:
> > > >     * This was used for ktcpsocket only, and the only remaining benefit
> > > >     of that was the KSslCertificateManager integration. This can easily
> > > >     be readded externally by implementing the Session::sslErrors
> > > >     callback and the Session::ignoreErrors function, as well as
> > > >     QSslSOckets default ca certificate functions. (We may want to extend
> > > >     the API there a bit).
> > > >     * KIO has a whole slew of unrelated dependencies so this was an
> > > >     important step to make KIMAP2 more selfcontained.
> > > 
> > > So is there some factory-like API that would allow us to provide our own
> > > KTcpSocket, falling back to QSslSocket by default? If not, would that be
> > > an acceptable change for you?
> > 
> > There is no such API. This would require an additional SSlSocket
> > abstraction layer to
> > hide the API differences between QSslSocket and KTcpSocket. Doable but I
> > don't understand why you'd want to do that if you can have all the
> > benefits that KTcpSocket  has (the KSslCertificateManager integration),
> > by just readding that externally.
> > 
> > Overall I hope we can avoid introducing that extra abstraction layer and
> > either:
> > * Give you what you need from KTcpSocket by allowing to easily build the
> > necessary integrations externally (on top of QSslSocket). As I said, I
> > think this is possible with minimal effort.
> > * or, Untie the whole connection setup from KIMAP, so it is completely
> > socket agnostic. This would be interesting I guess because it would open
> > doors to things like:
> >    * Have a trusted process that handles the authentication (signond).
> >    * Have the trusted process pass over the already authenticated socket
> >    to the process wanting to do some imap.
> >    * Have the untrusted process (imap resource), pass the socket to
> >    kimap so it can do some imap without ever having seen any sort of
> >    credentials.
> > 
> > Of course the latter strays further from the current design and is thus
> > a bit harder to implement.
> 
> I see. I now see that you mentioned that initially too, I just missed it 
> somehow. So KDE integration should not be a concern.

Jup, that should work out nicely.

> > 
> > > > * The login job has received an overhaul.
> > > > 
> > > >     * Removed the slightly confused EncryptionMode names that sometime
> > > >     mixed the excryption and the use of starttls (which are two largely
> > > >     unrelated things).
> > > >     * Cleaned up the login logic which was a very complex statemachine
> > > >     scattered accross different classes and is now only a complex
> > > >     statemachine scattered over fewer classes =(
> > > >     * Fixed a potential race-condition where we would send a CAPABILITY
> > > >     request before receiving the greeting, which some servers seem to
> > > >     handle anyways but some don't.
> > > >     * Removed the encryption negotation which is handled by QSslSocket
> > > >     according to the provided settings, and otherwise fails as it
> > > >     should.
> > > 
> > > Cool :-) The login part is indeed complicated. I still have a SASL2
> > > plugin
> > > that implements Google's XOAUTH authentication mechanism (basicall OAuth
> > > via IMAP), should probably try to upstream that to both KIMAPs :-)
> > 
> > That would indeed be interesting to see whether that fits into the
> > current login procedure,
> > or whether spawns some ideas for a better design to accommodate that as
> > well.
> 
> The XOAUTH2 process is very similar to regular LOGIN methods. You handle
> the OAuth2 authentication externally, and then you invoke LoginJob with your 
> username (email address) and password, where instead of your actual
> password 
> you just provide the OAuth2 Access token, so it maps onto the current
> design 
> well. The nasty stuff is than handled by the XOAUTH SASL2 plugin on the
> lower level.
> 

Ok, That sounds easy enough to add.

Cheers,
Christian