[Kde-pim] akonadinext update: entity processing pipelines in resources

[Aaron J. Seigo]

hey :)

hows that for a subject line full of jargon? ;)

in the akonadinext repo, sychnronizer processes are approaching some sort of 
early-stage completeness. they currently:

* accept connections from clients
* inform clients when the revision in the store changes[1]
* take commands from clients and respond with a task completion message[2]
* load Resource plugins to deal with storing data and synchronizing with the 
source
* manage processing pipelines for entities

it's that last part that i'm writing about here, actually. a pipeline is zero 
or more processing units (soon to be plugins) that (currently) sit in a chain 
and do some processing on entities[3] whenever they are created, modified or 
deleted. we will be using this to populate indexes, trigger full text 
indexing, applying client-side filters, spam/scam detection, etc. etc.

anything that can be / should be done to a given entity when it appears, 
changes or is removed will happen in these pipelines.

things left to do:

* make PipelineFilter pluggable so that it is easy for people to add new 
filters (including ones we don't ship ourselves)
* generate a configuration scheme which Pipeline can use to populate pipelines 
at runtime according to the user's wishes
* write a few PipelineFilter plugins that do some actually useful things that 
we can use in testing

currently, pipelines are just a simple one-after-the-other processing afair. 
It is set up already for asynchronous processing, however. Eventually I would 
like to allow filters to note that they can be parallelized, should be run in 
a separate thread, ??? ... mostly so that we can increase throughput.

as for safety: before a pipeline is started, the entity will be stored on disk 
in storage, so if something Goes Wrong(tm) and the synchronizer process 
crashes data will not be lost and the pipeline can pick up again. to make this 
even more robust, i need to implement pipeline checkpointing so it 
automatically skip over problematic filters (self-healing).

at this point, i'm very much open to ideas and suggestions for the pipeline 
feature. this is the easiest point in time to adjust the design and 
implementation. so speak up if this interests you! :) also: if you want to 
hack on these things, you are very much welcome to get your hands dirty with 
me. there is a lot to do, it isn't (yet ;) hard to make useful progress in the 
code base, ... come in, the water's nice!

[1] this is actually faked right now, waiting on further development of the 
Storage class and how we use it; but the synchronizer side of this is all done 

[2] in case you're wondering about efficiency there, on my laptop a client and 
server can exchange in excess of 800,000 messages of this sort per second. 
that includes the buffer creation overhead, transmission, reception...

[3] emails, calendar items, whatever.

-- 
Aaron J. Seigo
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.kde.org/pipermail/kde-pim/attachments/20141217/c1280d5d/attachment.sig>
-------------- next part --------------
_______________________________________________
KDE PIM mailing list kde-pim at kde.org
https://mail.kde.org/mailman/listinfo/kde-pim
KDE PIM home page at http://pim.kde.org/