[Kde-pim] New feature: Hypervisor

[Volker Krause]

Hi,

On Tuesday 01 October 2013 22:38:25 Daniel Vrátil wrote:
> On Monday 30 of September 2013 19:38:51 Alessandro Accardo wrote:
> > Hey guys,
> > I write this email because of a new feature for akonadi I'm trying to
> > develop according to dvratil on IRC. This paper below is a sort of
> > analysis
> > and somehow a technical documentation, but is incomplete.
> > I hope you guys can complete this and develop together this interesting
> > piece of software.
>
> first, thanks for working on this!
> 
> I gave it a big deal of thinking today (I had a very nice and very long walk
> to the city because I missed my bus :/) and, after rereading your proposal
> several times, I have a few ideas and comments.

indeed, thanks for tackling this!
 
> [Also, disclaimer: this email is ridiculously long]

sorry, similarly long reply below ;)

> > Akonadi Hypervisor
> 
> Do we really have to call it hypervisor? Sounds like we are doing some
> virtualization magic :) Can't we call it "Stop & Go" [0] ? :-)
> 
> >  1  Introduction
> > 
> > Akonadi Hypervisor will be a new feature of Akonadi server which is needed
> > for the concept of bringing up only the requested or mandatory resources
> > when akonadi is started from a client.
> > This mechanism introduces some new concepts for akonadi that will surely
> > improve its performances as well as consume less resources.
> > 
> >  2  Akonadi Resources and Agents
> > 
> > An akonadi resource that wants to be hypervisor-enabled, will have to:
> > * declare its managed MIME types (as an array of plain strings, better if
> > matching with constants or a mime types list file)
> 
> This is already defined in resource .desktop files
> 
> $ cat /usr/share/akonadi/agents/imapresource.desktop | grep "X-Akonadi"
> X-Akonadi-MimeTypes=message/rfc822
> X-Akonadi-Capabilities=Resource
> X-Akonadi-Identifier=akonadi_imap_resource
> 
> > * declare its managed types of items, if any
> 
> What exactly is "type of item"? Item is an item :-) And a type is specified
> by mime type.
> 
> > * register as a client of its needed resources/agents, if any
> 
> Right, this can be used to create dependencies between resources. Given a
> second thought, I 'm not sure whether we really need this.
> 
> > * register the needed sync events, introducing the concept of weak and
> > strong sync
> 
> This sound overly complicated and not necessary at all (see below).
> 
> >  2.1  Registering as a client of resources and agents
> > 
> > What it means to be “registered as a client” is, telling (via Dbus) to the
> > hypervisor which resources and agents a client needs to be fully
> > functional. A client can be whoever can access Dbus and send a message
> > standardized as a client registration for the akonadi hypervisor.
> 
> I don't think that has to be done over DBus - the client can send list of
> requested mimetypes during Akonadi session initialization, that is something
> like
> 
> * OK Akonadi Almost IMAP Server [PROTOCOL 34]
> 0 LOGIN client1 message/rfc822
> 0 OK User logged in
> ...
> ...
> 
> It will also require much less invasive changes to the clients. If a client
> sends "LOGIN" command without any mimetypes, we just start all resources -
> either because it means that client is interested in everything, or because
> the client is too old and can't tell us what it's interested in. However we
> have to somehow differentiate when resource is connecting to Akonadi server
> - there's no point in resource specifying what mime types it supports, so
> maybe it could just send a dummy mime type, like
> application/x-vnd.akonadi.resource.
> 
> 
> The resources are handled by akonadi_control, which has it's DBus interface
> in org.freedesktop.Akonadi /ResourceManager so you might just want to
> extend it with methods like startInstance(QString instanceId) and
> stopInstance(QString instanceId) that Akonadi server could use to control
> the instances.
> >  2.2  Extending the synchronization concept
> > 
> > Synchronization for resources is a plain routine where a resource queries
> > in a given way its data provider, aggregating and structuring this
> > results in order to pass them to the akonadi backend.
> > We want to extend this concept introducing the meaning of Weak
> > Synchronization and Strong Synchronization.
> > 
> >  2.2.1  Weak Synchronization
> > 
> > A weak sync is a type of operation that can be executed at a given time,
> > if
> > the resource is alive, without other concepts. It is mainly what happens
> > now, when akonadi is brought up after the sync period passed.
> > 
> >  2.2.2  Strong Synchronization
> > 
> > This is the brand new concept, that is well explained by an example. When
> > a
> > user chooses the Strong Sync a resource should schedule itself in the
> > hypervisor schedule process (via Dbus). This means that, if the resource
> > is
> > down and the schedule time arrives, the hypervisor brings up the resource
> > giving her the time to synchronize and then, when finished, if there are
> > no
> > clients who want to use it, simply shuts down this process, gaining
> > memory.
> 
> The fact the we are starting and shutting down resources on demand should
> not be exposed to clients *at all*. Simply when request to synchronize a
> collection owned by a resource that is currently not running arrives to
> Akonadi server, the server launches the resource and waits for the sync to
> finish. Then the server can decide whether to shut it down, or keep it
> running. Possibly, the resource could hint the server whether it's ready to
> be shut down, or whether it has some tasks accumulated that has yet to be
> replayed and thus delay the shutdown.
> 
> >  3  The Akonadi Hypervisor
> > 
> > The Akonadi Hypervisor is a process, started by the akonadi_control, that
> > acts as the core of that functionality. A core process is needed because
> > of
> > cleanness of the akonadi_control code, and because of the various actions
> > performed by the hypervisor itself that, if performed by akonadi_control
> > instead, can bring to a big unnecessary complexity in the code and
> > possibly
> > even to a lack of performance. Treating the hypervisor as a separate
> > process brings up even the possibility to shut down the new function if
> > not
> > needed or if not working, bringing akonadi_control to remain a robust
> > piece
> > of software, leaving it almost untouched.
> 
> The akonadi_control process does nothing most of the time, so I would not
> worry about performance at all. When it comes to code complexity - well, if
> you write it nicely, complex code is not a problem :-) I would actually
> prefer this to be part of akonadi_control or Akonadi server. After all,
> what we are trying to do here, is to reduce the amount of running
> processes. Having dedicated akonadi_hypervisor (did I mention I don't like
> the name? :-) sort of goes against it.
> 
> I think it's completely fine to have this incorporated into Akonadi server
> itself - once you are dealing with client requests, you still need to
> process them on the server and it makes no sense to just pass the request
> from Akonadi server to akonadi_hypervisor to pass it to akonadi_control :-)

I agree. The original reasoning behind the process split was how "dangerous" 
the data is the processes touch, not code complexity (process separation does 
not necessarily help with that ;-) ). Resources are at the biggest risk, being 
exposed to all kinds of invalid, broken or even malicious data from the evil 
Internet. Clients and agents only touch data that at least passed the 
resources and serializers, the Akonadi server only operates on opaque binary 
blobs, and the Akonadi control process doesn't even touch any untrusted data.

Integration of the "hypervisor" feature wont change that.

> > This process will do the following:
> > * listen to Dbus for incoming messages of registration from clients
> 
> As said above, this can be done through LOGIN command, no need for DBus
> 
> > * keep a timer thread with the Strong Synchronizations requested
> > * bring up or shut down those resources and agents needed or not by the
> > clients
> 
> Can by controlled by the Akonadi server itself through akonadi_control's
> DBus interface.
> 
> >  3.1  Registering clients
> > 
> > From the hypervisor point of view, registering clients means write up the
> > information of that clients to its configuration repository (either a
> > database, a rc or something else).
> 
> Why not just hold this in memory? It has to be sent by the client every time
> anyway.
> 
> > The needed (mandatory) information that we must be sure to write up to the
> > configuration are:
> > Client name: a name communicated by the client itself
> 
> Each new connection to server "session" has a unique session ID, so no
> problem here.
> 
> > MIME Types: requested MIME types that the client needs to work properly
> 
> Would be send as part of LOGIN command, can be maintained in the session.
> 
> > Types of items: requested types of items that the client needs to work
> > properly
> > 
> >  3.2  Capability
> > 
> > The capability is in fact a property of the clients, but is treated here
> > because it has to be communicated by the clients to the hypervisor, via
> > Dbus.
> 
> I disagree. It's a capability of the server. We just need a bit more
> information from clients, but otherwise this should be completely hidden
> from clients.
> 
> 
> 
> 
> All in all pretty good proposal, thanks for it! One note however: we need a
> different approach to resources and to agents, see why:
> 
> In your proposal you suggest handling agents and resources as one, but that
> would not work well:  resources are started when there's a client working
> with their data, while agents have to be started based on some outside
> condition (there are no clients working directly with agents), which makes
> a big difference.
> 
> Example:
> 
> We want Nepomuk Feeder Agent to be running only when Nepomuk is running (and
> we don't care what resources are or are not running).
> 
> One possible solution how to deal with this is something I call "hooks"
> (which is not a good name, but I'm very bad at naming stuff). Hooks would
> be little plugins loaded by Akonadi server. They would simply have
> start(QString &instance) and stop(QString &instance) signals. So if we use
> our example again, hook for Akonadi Nepomuk Feeder Agent would emit
> start("akonadi_nepomuk_feeder") signal, when it detects that Nepomuk has
> been started and would emit stop("akonadi_nepomuk_feeder") when Nepomuk
> server disappears. Based on the signal, Akonadi server would use
> akonadi_control DBus interface to have the respective agent started or
> stopped.
> 
> I'm not saying that this solution cannot share some code with your proposed
> solution for resources, but the concept needs to be a bit different, since
> the concept of agents and resources is a bit different.
> 
> Second problem with agents: some agents have a configuration UI used by
> client apps - we can use the New Mail Notifier Agent as an example. In
> KMail, clicking on a button in accounts settings dialog, a notification
> configuration dialog is brought up. The dialog is actually owned by the
> agent and when you click "OK", the agent can immediately apply the
> configuration. If the agent would not be running then, well, it would not
> be able to show the configuration :-) Proposed solution would be to move
> such configuration into standalone KCM modules, which would work
> independently of whether the agent is running. The "hook" for this agent
> would just watch the configuration file and when it changes, it would start
> (or stop) the agent, depending on what content of the file is - so when
> user disables notifications, the hook would emit the stop signal as there's
> no need for having notifier agent running if it won't ever send any
> notification.
> 
> Another thing to consider (and that will probably require some larger
> changes) is how to tell clients about all resources. Right now if a client
> wants to list all available resources and agents, it simply lists all
> services with org.freedesktop.Akonadi.Agent.* interface.

are you sure? IIRC agent listing is done via akonadi_control's AgentManager D-
Bus interface (cf. agentInstances()).

> If we will be
> shutting down these resources, they would not appear on DBus and thus could
> not be listed, so we need a different way to list them - probably through
> another extension to Akonadi protocol or Akonadi's DBus interface that
> would allow listing all resources (even those shut down, with a new status
> like "Suspended" or something) and a centralized way to configure them - so
> probably again, instead of calling
> org.freedesktop.Akonadi.Agent.Control.configure(), there would have to be a
> new method like
> org.freedesktop.Akonadi.ResourceManager.configure(QString instance)
> implemented no the server, where the Akonadi server would check whether the
> instance is running (and start it if necessary) and call it's configure()
> method.

agentInstanceConfigure(QString,WId) in akonadi_control? :)

This approach would work for anything using D-Bus as a "resume trigger", see 
below for a more aggressive version of this.

> Ok, I think I've mentioned everything I wanted for now (except for one thing
> I can't remember right now, so I'll follow up when I remember).
> 
> Also if anyone else made it down here, please share your thoughts and ideas
> on this. I think it's a pretty good idea that can help us not to be the
> "most resources-hungry" project on embedded devices, and to be less hungry
> on desktops too. Judging from general feedback, I think many people will
> appreciate that :-)

Ok, here's my thoughts on this:


* Focus on agents first, not resources

- as long as eg. the nepomuk feeder or the new mail notifier are running, 
you'd hardly manage to shut down resources
- resources are a special case of agents, so a solution for agents should 
cover most of what's needed already
- resources will need server side change recording for safe suspension first, 
which is another big task


* Suspending agents

That's the easy part, give agents a "suspend()" method that tells 
akonadi_control it's allowed to shut them down. The decision can be made by 
the agent itself (or also the resource base class for example), with 
akonadi_control having the final say (to avoid quick suspend/resume loops for 
example).


* Resuming agents

The tricky part is triggering this when needed. From the discussion so far we 
seem to have the following scenarios:

(1) D-Bus calls: This includes sync requests for resources or changes to the 
agent configuration. As Dan outlined we can intercept the well-known calls by 
the already existing redirection through akonadi_control, and start the agent 
on demand.

To cover even custom interfaces (e.g. the send later agent), there's the 
following systemd-inspired crazy idea: Before shutting down an agent we 
"clone" it's D-Bus interface in akonadi_control, and use that for on-demand 
starting similar to the approach for well-known methods described above. 
qdbusviewer shows you can completely introspect the interface, 
QMetaObjectBuilder & friends allow to create custom QMetaObjects, which then 
could be exported via D-Bus again. Needs a bit of research to see if that's 
actually feasible, and where the limitations are, but it would be a nicely 
generic solution to any D-Bus call :)

(2) Availability of an external D-Bus service (Nepomuk).

(3) Availability of network (resources).

(4) Changes to entities owned by a resource. Easy, as this goes through the 
Akonadi server, adding resource resuming if the server-side change recorder 
starts to fill up should be straightforward.

(5) Combinations of the above: (Nepomuk feeder: (1) + (2), networked 
resources: (3) + (4)). This is relevant to avoid suspend/resume loops if for 
example being offline while editing items in your IMAP account.

Did I miss any resume trigger not covered by any of the above?

One solution to this (and obviously the most generic one) is the hooks 
proposed by Dan. However, loading arbitrary third-party code into 
akonadi_control is kinda ugly regarding the process separation concept 
outlined above. So can we get around it? (1) and (4) would be fully 
transparent, (2), (3) and (5) need some more thought.

(2) and (3) could be modeled by "D-Bus conditions", ie. simple expressions on 
service names or property values on D-Bus. That can be done without executing 
custom code in akonadi_control, it's merely a textual declaration. The 
.desktop file would be the obvious place for this (e.g. feeder agent requires 
the "org.kde.NepomukServer" service to be present), but that's not good enough 
for network availability (a localhost IMAP server would not be affected by 
being offline). That would require a runtime API to specify "resume 
conditions".

(5) ideally is just the AND combination of all "resume conditions" ((2), (3)), 
and if that evaluates to true on a "resume trigger" ((1) or (4)), we actually 
resume.


* Compatibility & Safety

The above approach would be largely backward compatible:
- agents not calling suspend() would not be affected at all
- resources could have suspend() calls built-in as part of the resource 
scheduler, but without "resume conditions" specified (ie. resume always 
possible) would be waken up as soon as something interacts with their data 
(assuming server side change recording of course)
- "resume triggers" are fully transparent
- minimal code change required to add suspend support to non-resource agents, 
and no code change for most resources

Stuff like the mail filter agent would only shut down if no filter is 
configured, not if there are, but no mails are coming in at the moment. IMHO 
that's an acceptable compromise between reliability and performance/overhead.


regards,
Volker
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 190 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.kde.org/pipermail/kde-pim/attachments/20131003/b734d9f0/attachment.sig>
-------------- next part --------------
_______________________________________________
KDE PIM mailing list kde-pim at kde.org
https://mail.kde.org/mailman/listinfo/kde-pim
KDE PIM home page at http://pim.kde.org/