[Kde-pim] akonadinext update: entity processing pipelines in resources

[Milian Wolff]

On Wednesday 17 December 2014 15:06:35 Christian Mollekopf wrote:
> On Wednesday 17 December 2014 14.49:53 Milian Wolff wrote:
> > On Wednesday 17 December 2014 11:39:10 Aaron J. Seigo wrote:
> > > hey :)
> > > 
> > > hows that for a subject line full of jargon? ;)
> > > 
> > > in the akonadinext repo, sychnronizer processes are approaching some
> > > sort
> > > of early-stage completeness. they currently:
> > > 
> > > * accept connections from clients
> > > * inform clients when the revision in the store changes[1]
> > > * take commands from clients and respond with a task completion
> > > message[2]
> > > * load Resource plugins to deal with storing data and synchronizing with
> > > the source
> > > * manage processing pipelines for entities
> > > 
> > > it's that last part that i'm writing about here, actually. a pipeline is
> > > zero or more processing units (soon to be plugins) that (currently) sit
> > > in
> > > a chain and do some processing on entities[3] whenever they are created,
> > > modified or deleted. we will be using this to populate indexes, trigger
> > > full text indexing, applying client-side filters, spam/scam detection,
> > > etc.
> > > etc.
> > > 
> > > anything that can be / should be done to a given entity when it appears,
> > > changes or is removed will happen in these pipelines.
> > > 
> > > things left to do:
> > > 
> > > * make PipelineFilter pluggable so that it is easy for people to add new
> > > filters (including ones we don't ship ourselves)
> > > * generate a configuration scheme which Pipeline can use to populate
> > > pipelines at runtime according to the user's wishes
> > > * write a few PipelineFilter plugins that do some actually useful things
> > > that we can use in testing
> > > 
> > > currently, pipelines are just a simple one-after-the-other processing
> > > afair. It is set up already for asynchronous processing, however.
> > > Eventually I would like to allow filters to note that they can be
> > > parallelized, should be run in a separate thread, ??? ... mostly so that
> > > we can increase throughput.
> > 
> > This, imo, will kill user-configuration. You do not want to burden the
> > user
> > with a GUI where he can define dependencies etc. pp.
> > 
> > Also, I cannot think of any common use-case of mail filtering that could
> > be
> > parallelized for a single mail:
> > 
> > a) first, "move" filters are checked, such as spam filters which either
> > discard the mail or put it into a sub folder, or mailing list filters,
> > which put the mails into a folder for the given list. when any filter is
> > met here, the chain is stopped, thus it cannot be parallelized
> > b) if the final place is found, and it is a folder that we want to have
> > indexed, we feed it over to e.g. baloo. again, not something you can do in
> > parallel, as you don't want to index spam mails, and also want to know the
> > final place of the mail.
> > 
> > What other, _common_ usecase do you think of that would benefit from the
> > additional design overhead?
> 
> "filter" in this context are not what we currently have as client-side
> filtering. It's rather a "processor" if you will, that get's processed as
> new or modified entites are processed.
> 
> See also:
> https://community.kde.org/KDE_PIM/Akonadi_Next/Terminology
> 
> Filter could be used for:
> * indexing
> * detecting spam
> * client-side filtering (which is what you meant I think)
> * ....
> 
> I basically allows us to plug in pieces of functionality that we can
> guarantee get processed before an entity officially enters the system.
> 
> So most of these filters will be fixed by the configuration shipped with the
> resource, not something the user can adjust. Some filter may be optional or
> react to user configuration such as client-side filtering, or optional
> full- text indexing.

The rest of my mail still stands as-is though. What do you think of that can 
potentially be parallelized? What, besides client-side filtering, would be 
configurable by the user?

Bye
-- 
Milian Wolff
mail at milianw.de
http://milianw.de
_______________________________________________
KDE PIM mailing list kde-pim at kde.org
https://mail.kde.org/mailman/listinfo/kde-pim
KDE PIM home page at http://pim.kde.org/