[Kde-pim] PIM Sprint report: Akonadi Next

Wed Nov 26 14:07:00 GMT 2014

On Wednesday 26 November 2014 14.17:41 Milian Wolff wrote:
> On Wednesday 26 November 2014 13:16:49 Daniel Vrátil wrote:
> > On Wednesday 26 of November 2014 12:36:56 Milian Wolff wrote:
> > > On Wednesday 26 November 2014 10:46:18 Daniel Vrátil wrote:
> > > > Hi all,
> > > > 
> > > > the PIM sprint is over and we have some exciting future ahead of us!
> > > > 
> > > > Let's start with the largest change: Christian and Aaron came up with
> > > > a
> > > > proposal for redesign of Akonadi using modern reactive design
> > > > patterns.
> > > > We
> > > > polished and sorted out a lot of details there, and I'm really excited
> > > > about the plan, as it will allow Akonadi to benefit from modern
> > > > technologies, making the codebase simpler, faster and more robust (I
> > > > guess that's what you guys said about the current Akonadi too ;-)). We
> > > > also hope to attract new contributors by using the new "cool" stuff.
> > > > Big
> > > > thanks to Till for his insights into current Akonadi design.
> > > 
> > > Can you share some more about the ideas you have? This teaser above has
> > > me
> > > hooked, but it doesn't really say that much about what you guys are up
> > > to
> > > 
> > > :)
> > 
> > Basically the new design absents central server in favor of using
> > per-resource storage. Each resource would maintain it's own key-value
> > store
> > (allowing each resource to store data in a way that is efficient for them
> > to work with) and would provide an implementation of standardized
> > interface
> > to access this storage directly. Clients would be allowed direct read-only
> > access to these storages, while resources are the only one with write
> > access. Internally flatbuffers would be used to provide access to the data
> > in super-efficient way (lazy-loading, memory mapping, all the fancy
> > stuff).
> 
> OK, cool. That sounds good to me. Actually it sounds extremely similar to
> what I have planned for the future of KDevelop as well. The global cache we
> use there is suboptimal, and now that we work on Clang support, it doesn't
> really fit right. So I'm looking forward to see your work on this in
> Akonadi, maybe I can even reuse parts of this stuff then :)
> 
> One question: Why flatbuffers? I'm not arguing against it, I just would like
> to know why that, and not one of the other cool kids on the block, most
> notably Cap'n'Proto. I only read this once:
> http://kentonv.github.io/capnproto/news/2014-06-17-capnproto-flatbuffers-sbe
> .html
> 
> So if anyone did an educated decision, I'd like to know the reasoning. The
> slides below seem to indicate that multiple things where researched.
> 

We looked mostly at Cap'n'Proto and flatbuffers, and they generally both solve 
the same problems we need solved. Both provide backwards compatibility with 
default values, easy schema definitions and zero-copy loading where we can 
directly mmap the file.

We opted for flatbuffers as proof of concept mostly for portability reasons. 
It also does seem to have a larger contributor base, and doesn't really have 
and drawbacks for us.

> Also, from a lessons-learned POV, I really urge you all to somehow wrap the
> serialization code such that it can be replaced easily later on. I know that
> I'd love to have that in KDevelop. Now I'll have to refactor tons of code
> to use anything else...
> 

Yup, we'll definitely want to wrap that =) And we definitely won't leak any of 
those details to the applications themselves.

> > Resources would implement pipelines allowing some pre-processing of newly
> > incoming data before storing them persistently (think mail filtering,
> > indexing, new mail notifications etc.). Inter-resource communication (for
> > example to perform inter-resource move or copy), and client->resource
> > communication would be done through a binary protocol based on what we
> > have
> > now in Akonadi. This design also gives us on-demand start/stop of
> > resources
> > for free. Something that requires ridiculous amount of work to make work
> > with the current design.
> 
> Cool. But if you use FlatBuffers or anything else, why not use that instead
> of a custom binary protocol?
> 

Applications with directly access the data through an abstraction layer. So 
data-access no longer requires a protocol at all and we can access the buffers 
ideally zero-copy.

We still do require a slim protocol to send modifications to resources, for 
which I intend to use flatbuffers as well.

> > API-wise, while we can't completely get rid of the "imperative" API of
> > having jobs, the core method to provide access to data would be through
> > models. Making use of storage data versioning, on update the model simply
> > requests changes between current and last revision of the stored data.
> > This
> > should prevent us from ending up with overcomplicated beast-models like
> > ETM.
> > 
> > This design is rather different from the current one, so we won't probably
> > be able to reuse much of the existing code, hence the long timeframe.
> > 
> > Aaron has also shared some slides [0] that Christian presented on the
> > sprint regarding this change
> > 
> > [0] https://plus.google.com/+AaronSeigo/posts/Xt7HNZJV5UJ
> 
> Interesting, thanks. Slide 17 has me worried though. How do you intend to
> make this thread/multi-process safe, when you have read/write access from
> the resource and _direct_ read access from the client? That should probably
> be indirected to make each resource be able to use its own storage and
> centralize locking stuff (at least no multi-process issues then).
> 

We build this around a key-value store that is multi-process and thread safe. 
Since only the resource can write to it's store and we can have as many 
concurrent readers as we want this shouldn't be a problem.

> > > > Implementing these changes and stabilizing the project will take lots
> > > > of
> > > > time. On Akademy I mentioned that my plans I had with Akonadi would
> > > > take
> > > > roughly a year to implement. I expect that this will take at least the
> > > > same
> > > > amount of time, maybe even more, since I'll have considerably less
> > > > time
> > > > to
> > > > work on PIM in the near future. I gave some thoughts to the current
> > > > situation and I think that it might be a good idea to implement some
> > > > of
> > > > my
> > > > ideas on the current Akonadi master, and release as soon as possible,
> > > > so
> > > > that people can get at least some bug fixes and performance
> > > > improvements
> > > > while we work on the "Akonadi Next". Since KDE PIM is basically ported
> > > > to
> > > > Qt 5/KF 5 (with kdelibs4support, but that does not really matter) and
> > > > seems
> > > > to work (more or less), we could try to release from master with next
> > > > KDE
> > > > Applications (15.04 or whatever the version turns out to be) without
> > > > any
> > > > guarantees and promises of API and ABI compatibility of the
> > > > Frameworks.
> > > > This is necessary, because we can't really wait with release until we
> > > > can
> > > > give this promise, as there are other projects depending on Akonadi
> > > > (Plasma, KTp, ....) and we are effectively blocking them. Once we
> > > > decide
> > > > that the new Akonadi is ready, we should be able to simply switch from
> > > > the
> > > > old one to the new one and work towards stable API/ABI of the
> > > > libraries.
> > > > Laurent, and other PIM maintainers: opinions?
> > > 
> > > Releasing a library without any ABI guarantees is like not releasing a
> > > library at all, no?
> > 
> > No. If you don't release a library, you can't have released applications
> > using it. If you release it, you can have applications using it. The fact
> > that there is not ABI guarantees is not a problem, you can still make sure
> > that only the applications you want (you have under control) use it (let's
> > say by not installing AkonadiConfig.cmake, but shipping FindAkonadi.cmake
> > in the application.
> 
> So plasma will be coupled to Akonadi, and a change of ABI in Akonadi will
> trigger a rebuild of Plasma. Not nice, distributions will hate this.
> 
> > > What is Plasma, Ktp, ... actually using from Akonadi?
> > 
> > Plasma needs it for the calendar applet (it's getting a bit annoying that
> > on almost every promo screenshot we are showing off our new awesome
> > beautiful calendar applet without admitting it actually can't show
> > anything except for current date). KTp uses KPeople, which has an Akonadi
> > backend to get more info about metacontacts.
> > 
> > > Could maybe a small shim library be created, specifically for that
> > > purpose,
> > > which can be shipped with an ABI guarantee?
> > 
> > Not really.
> 
> Why? I'm still not convinced. For the calender e.g. create a small library
> that can be used by the Plasma applet and be done with it.
> 

I guess the only question is how big that API has to be in order for it not to 
become a maintenance burden. I also think it is reasonable for us to declare 
for the few external users what API they can use, and to phase out that API 
once we know we ported all users. That means we don't give a general ABI 
guarantee, and keep the right to phase this out once no users of the legacy 
API remain (even if we leave the skeleton without functionality in place for 
ABI reasons).

Cheers,
Christian
_______________________________________________
KDE PIM mailing list kde-pim at kde.org
https://mail.kde.org/mailman/listinfo/kde-pim
KDE PIM home page at http://pim.kde.org/