[Kde-pim] PIM Sprint report: Akonadi Next

Wed Nov 26 13:33:43 GMT 2014

On Wednesday 26 of November 2014 14:17:41 Milian Wolff wrote:
> On Wednesday 26 November 2014 13:16:49 Daniel Vrátil wrote:
> > On Wednesday 26 of November 2014 12:36:56 Milian Wolff wrote:
> > > On Wednesday 26 November 2014 10:46:18 Daniel Vrátil wrote:
> > > > Hi all,
> > > > 
> > > > the PIM sprint is over and we have some exciting future ahead of us!
> > > > 
> > > > Let's start with the largest change: Christian and Aaron came up with
> > > > a
> > > > proposal for redesign of Akonadi using modern reactive design
> > > > patterns.
> > > > We
> > > > polished and sorted out a lot of details there, and I'm really excited
> > > > about the plan, as it will allow Akonadi to benefit from modern
> > > > technologies, making the codebase simpler, faster and more robust (I
> > > > guess that's what you guys said about the current Akonadi too ;-)). We
> > > > also hope to attract new contributors by using the new "cool" stuff.
> > > > Big
> > > > thanks to Till for his insights into current Akonadi design.
> > > 
> > > Can you share some more about the ideas you have? This teaser above has
> > > me
> > > hooked, but it doesn't really say that much about what you guys are up
> > > to
> > > 
> > > :)
> > 
> > Basically the new design absents central server in favor of using
> > per-resource storage. Each resource would maintain it's own key-value
> > store
> > (allowing each resource to store data in a way that is efficient for them
> > to work with) and would provide an implementation of standardized
> > interface
> > to access this storage directly. Clients would be allowed direct read-only
> > access to these storages, while resources are the only one with write
> > access. Internally flatbuffers would be used to provide access to the data
> > in super-efficient way (lazy-loading, memory mapping, all the fancy
> > stuff).
> 
> OK, cool. That sounds good to me. Actually it sounds extremely similar to
> what I have planned for the future of KDevelop as well. The global cache we
> use there is suboptimal, and now that we work on Clang support, it doesn't
> really fit right. So I'm looking forward to see your work on this in
> Akonadi, maybe I can even reuse parts of this stuff then :)
> 
> One question: Why flatbuffers? I'm not arguing against it, I just would like
> to know why that, and not one of the other cool kids on the block, most
> notably Cap'n'Proto. I only read this once:
> http://kentonv.github.io/capnproto/news/2014-06-17-capnproto-flatbuffers-sbe
> .html
> 
> So if anyone did an educated decision, I'd like to know the reasoning. The
> slides below seem to indicate that multiple things where researched.

I haven't done any investigation, so this question will best be answered by 
Aaron or Christian.

The one thing that I can read from the feature matrix is that FlatBuffers, 
unlike Cap'n'Proto supports MSVC, and also works on Android and iOS(?), paving 
our path for PIM domination across all platforms :-D No, seriously - Windows 
support is the important bit here.

> 
> Also, from a lessons-learned POV, I really urge you all to somehow wrap the
> serialization code such that it can be replaced easily later on. I know that
> I'd love to have that in KDevelop. Now I'll have to refactor tons of code
> to use anything else...
> 
> > Resources would implement pipelines allowing some pre-processing of newly
> > incoming data before storing them persistently (think mail filtering,
> > indexing, new mail notifications etc.). Inter-resource communication (for
> > example to perform inter-resource move or copy), and client->resource
> > communication would be done through a binary protocol based on what we
> > have
> > now in Akonadi. This design also gives us on-demand start/stop of
> > resources
> > for free. Something that requires ridiculous amount of work to make work
> > with the current design.
> 
> Cool. But if you use FlatBuffers or anything else, why not use that instead
> of a custom binary protocol?

Depends on how easily FlatBuffers can be used for inter-process communication, 
and if more free-formed structure wouldn't work better for the protocol. But 
you are most probably right.

> 
> > API-wise, while we can't completely get rid of the "imperative" API of
> > having jobs, the core method to provide access to data would be through
> > models. Making use of storage data versioning, on update the model simply
> > requests changes between current and last revision of the stored data.
> > This
> > should prevent us from ending up with overcomplicated beast-models like
> > ETM.
> > 
> > This design is rather different from the current one, so we won't probably
> > be able to reuse much of the existing code, hence the long timeframe.
> > 
> > Aaron has also shared some slides [0] that Christian presented on the
> > sprint regarding this change
> > 
> > [0] https://plus.google.com/+AaronSeigo/posts/Xt7HNZJV5UJ
> 
> Interesting, thanks. Slide 17 has me worried though. How do you intend to
> make this thread/multi-process safe, when you have read/write access from
> the resource and _direct_ read access from the client? That should probably
> be indirected to make each resource be able to use its own storage and
> centralize locking stuff (at least no multi-process issues then).

The key-value store that Aaron and Christian have chosen (LMDB, if I remember 
it correctly) handles this internally. Don't ask me how..:-)

> > > > Implementing these changes and stabilizing the project will take lots
> > > > of
> > > > time. On Akademy I mentioned that my plans I had with Akonadi would
> > > > take
> > > > roughly a year to implement. I expect that this will take at least the
> > > > same
> > > > amount of time, maybe even more, since I'll have considerably less
> > > > time
> > > > to
> > > > work on PIM in the near future. I gave some thoughts to the current
> > > > situation and I think that it might be a good idea to implement some
> > > > of
> > > > my
> > > > ideas on the current Akonadi master, and release as soon as possible,
> > > > so
> > > > that people can get at least some bug fixes and performance
> > > > improvements
> > > > while we work on the "Akonadi Next". Since KDE PIM is basically ported
> > > > to
> > > > Qt 5/KF 5 (with kdelibs4support, but that does not really matter) and
> > > > seems
> > > > to work (more or less), we could try to release from master with next
> > > > KDE
> > > > Applications (15.04 or whatever the version turns out to be) without
> > > > any
> > > > guarantees and promises of API and ABI compatibility of the
> > > > Frameworks.
> > > > This is necessary, because we can't really wait with release until we
> > > > can
> > > > give this promise, as there are other projects depending on Akonadi
> > > > (Plasma, KTp, ....) and we are effectively blocking them. Once we
> > > > decide
> > > > that the new Akonadi is ready, we should be able to simply switch from
> > > > the
> > > > old one to the new one and work towards stable API/ABI of the
> > > > libraries.
> > > > Laurent, and other PIM maintainers: opinions?
> > > 
> > > Releasing a library without any ABI guarantees is like not releasing a
> > > library at all, no?
> > 
> > No. If you don't release a library, you can't have released applications
> > using it. If you release it, you can have applications using it. The fact
> > that there is not ABI guarantees is not a problem, you can still make sure
> > that only the applications you want (you have under control) use it (let's
> > say by not installing AkonadiConfig.cmake, but shipping FindAkonadi.cmake
> > in the application.
> 
> So plasma will be coupled to Akonadi, and a change of ABI in Akonadi will
> trigger a rebuild of Plasma. Not nice, distributions will hate this.

It's just temporary...for real. Not like "temporary workarounds" :-)

> 
> > > What is Plasma, Ktp, ... actually using from Akonadi?
> > 
> > Plasma needs it for the calendar applet (it's getting a bit annoying that
> > on almost every promo screenshot we are showing off our new awesome
> > beautiful calendar applet without admitting it actually can't show
> > anything except for current date). KTp uses KPeople, which has an Akonadi
> > backend to get more info about metacontacts.
> > 
> > > Could maybe a small shim library be created, specifically for that
> > > purpose,
> > > which can be shipped with an ABI guarantee?
> > 
> > Not really.
> 
> Why? I'm still not convinced. For the calender e.g. create a small library
> that can be used by the Plasma applet and be done with it.

I'm afraid that writing and maintaining another layer of code would just be 
too much work for me. I need this to move forward fast...ish.

> 
> > > Then the rest can be changed at will and these "external" apps can live
> > 
> > happily.
> > 
> > No external applications. Instead we restrict the usage to applications we
> > control and where we can make sure that they always require a compatible
> > library (through soname bumps, version requirements...)
> 
> Plasma is still external from an Akonadi POV.

Right, but still "under our control" in a sense, unlike completely 3rd-party 
projects.

> 
> > > For KDEPIM and the other
> > > "heavy lifters", which will definitely require more access to Akonadi
> > > API,
> > > that is not such a big deal. Since that will probably be released
> > > together
> > > with Akonadi anyways, no?
> > 
> > Yep.
> > 
> > > And it was/is planned to make more stuff internal
> > > anyways, right?
> > 
> > Only Akonadi server is internalized. What is now in kdepimlibs.git/akonadi
> > will all remain public.
> 
> OK, thanks.
> 
> > > > We also discussed some more repositories reshuffling:
> > > > 
> > > > kdepimlibs:
> > > > * merge akonadi-{contacts,calendar,mime,notes,socialutils} to
> > > > akonadi-extras * merge gpgme++ and qgpgme
> > > > * finish killing kpimutils
> > > > 
> > > > kdepim-runtime:
> > > > * agents -> akonadi-resources
> > > > * resources -> akonadi-resources
> > > > * plugins -> akonadi-extras
> > > > * qml -> akonadi (src/declarative)
> > > > * resourcetester -> akonadi-resources
> > > > * kioslave -> could go to kio-extras, or akonadi-extras
> > > > * migration -> akonadi-migration
> > > > 
> > > > kdepim:
> > > > - no changes atm
> > > > 
> > > > I'm not mentioning the other topics like VDG and QML API as I wasn't
> > > > there
> > > > or wasn't paying enough attention :), but I hope others who were there
> > > > will
> > > > send their summary too.
> > > 
> > > Sounds like an interesting sprint! Rock on guys

-- 
Daniel Vrátil | dvratil at redhat.com | dvratil on #kde-devel, #kontact, #akonadi
Software Engineer - KDE Desktop Team, Red Hat Inc.

GPG Key: 0xC59D614F6F4AE348
Fingerprint: 4EC1 86E3 C54E 0B39 5FDD B5FB C59D 614F 6F4A E348
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 473 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.kde.org/pipermail/kde-pim/attachments/20141126/f31a449c/attachment.sig>
-------------- next part --------------
_______________________________________________
KDE PIM mailing list kde-pim at kde.org
https://mail.kde.org/mailman/listinfo/kde-pim
KDE PIM home page at http://pim.kde.org/