[Kde-pim] New feature: Hypervisor
Daniel Vrátil
dvratil at redhat.com
Tue Oct 1 21:38:25 BST 2013
On Monday 30 of September 2013 19:38:51 Alessandro Accardo wrote:
> Hey guys,
> I write this email because of a new feature for akonadi I'm trying to
> develop according to dvratil on IRC. This paper below is a sort of analysis
> and somehow a technical documentation, but is incomplete.
> I hope you guys can complete this and develop together this interesting
> piece of software.
>
> Bye,
> kLeZ-hAcK.
Hi,
first, thanks for working on this!
I gave it a big deal of thinking today (I had a very nice and very long walk
to the city because I missed my bus :/) and, after rereading your proposal
several times, I have a few ideas and comments.
[Also, disclaimer: this email is ridiculously long]
> Akonadi Hypervisor
Do we really have to call it hypervisor? Sounds like we are doing some
virtualization magic :) Can't we call it "Stop & Go" [0] ? :-)
>
> 1 Introduction
> Akonadi Hypervisor will be a new feature of Akonadi server which is needed
> for the concept of bringing up only the requested or mandatory resources
> when akonadi is started from a client.
> This mechanism introduces some new concepts for akonadi that will surely
> improve its performances as well as consume less resources.
>
> 2 Akonadi Resources and Agents
> An akonadi resource that wants to be hypervisor-enabled, will have to:
> * declare its managed MIME types (as an array of plain strings, better if
> matching with constants or a mime types list file)
This is already defined in resource .desktop files
$ cat /usr/share/akonadi/agents/imapresource.desktop | grep "X-Akonadi"
X-Akonadi-MimeTypes=message/rfc822
X-Akonadi-Capabilities=Resource
X-Akonadi-Identifier=akonadi_imap_resource
> * declare its managed types of items, if any
What exactly is "type of item"? Item is an item :-) And a type is specified by
mime type.
> * register as a client of its needed resources/agents, if any
Right, this can be used to create dependencies between resources. Given a
second thought, I 'm not sure whether we really need this.
> * register the needed sync events, introducing the concept of weak and
> strong sync
This sound overly complicated and not necessary at all (see below).
> 2.1 Registering as a client of resources and agents
> What it means to be “registered as a client” is, telling (via Dbus) to the
> hypervisor which resources and agents a client needs to be fully functional.
> A client can be whoever can access Dbus and send a message standardized as
> a client registration for the akonadi hypervisor.
I don't think that has to be done over DBus - the client can send list of
requested mimetypes during Akonadi session initialization, that is something
like
* OK Akonadi Almost IMAP Server [PROTOCOL 34]
0 LOGIN client1 message/rfc822
0 OK User logged in
...
...
It will also require much less invasive changes to the clients. If a client
sends "LOGIN" command without any mimetypes, we just start all resources -
either because it means that client is interested in everything, or because
the client is too old and can't tell us what it's interested in. However we
have to somehow differentiate when resource is connecting to Akonadi server -
there's no point in resource specifying what mime types it supports, so maybe
it could just send a dummy mime type, like
application/x-vnd.akonadi.resource.
The resources are handled by akonadi_control, which has it's DBus interface in
org.freedesktop.Akonadi /ResourceManager so you might just want to extend it
with methods like startInstance(QString instanceId) and stopInstance(QString
instanceId) that Akonadi server could use to control the instances.
> 2.2 Extending the synchronization concept
> Synchronization for resources is a plain routine where a resource queries in
> a given way its data provider, aggregating and structuring this results in
> order to pass them to the akonadi backend.
> We want to extend this concept introducing the meaning of Weak
> Synchronization and Strong Synchronization.
>
> 2.2.1 Weak Synchronization
> A weak sync is a type of operation that can be executed at a given time, if
> the resource is alive, without other concepts. It is mainly what happens
> now, when akonadi is brought up after the sync period passed.
>
> 2.2.2 Strong Synchronization
> This is the brand new concept, that is well explained by an example. When a
> user chooses the Strong Sync a resource should schedule itself in the
> hypervisor schedule process (via Dbus). This means that, if the resource is
> down and the schedule time arrives, the hypervisor brings up the resource
> giving her the time to synchronize and then, when finished, if there are no
> clients who want to use it, simply shuts down this process, gaining memory.
The fact the we are starting and shutting down resources on demand should not
be exposed to clients *at all*. Simply when request to synchronize a
collection owned by a resource that is currently not running arrives to
Akonadi server, the server launches the resource and waits for the sync to
finish. Then the server can decide whether to shut it down, or keep it running.
Possibly, the resource could hint the server whether it's ready to be shut
down, or whether it has some tasks accumulated that has yet to be replayed and
thus delay the shutdown.
> 3 The Akonadi Hypervisor
> The Akonadi Hypervisor is a process, started by the akonadi_control, that
> acts as the core of that functionality. A core process is needed because of
> cleanness of the akonadi_control code, and because of the various actions
> performed by the hypervisor itself that, if performed by akonadi_control
> instead, can bring to a big unnecessary complexity in the code and possibly
> even to a lack of performance. Treating the hypervisor as a separate
> process brings up even the possibility to shut down the new function if not
> needed or if not working, bringing akonadi_control to remain a robust piece
> of software, leaving it almost untouched.
The akonadi_control process does nothing most of the time, so I would not
worry about performance at all. When it comes to code complexity - well, if
you write it nicely, complex code is not a problem :-) I would actually prefer
this to be part of akonadi_control or Akonadi server. After all, what we are
trying to do here, is to reduce the amount of running processes. Having
dedicated akonadi_hypervisor (did I mention I don't like the name? :-) sort of
goes against it.
I think it's completely fine to have this incorporated into Akonadi server
itself - once you are dealing with client requests, you still need to process
them on the server and it makes no sense to just pass the request from Akonadi
server to akonadi_hypervisor to pass it to akonadi_control :-)
> This process will do the following:
> * listen to Dbus for incoming messages of registration from clients
As said above, this can be done through LOGIN command, no need for DBus
> * keep a timer thread with the Strong Synchronizations requested
> * bring up or shut down those resources and agents needed or not by the
> clients
Can by controlled by the Akonadi server itself through akonadi_control's DBus
interface.
> 3.1 Registering clients
> From the hypervisor point of view, registering clients means write up the
> information of that clients to its configuration repository (either a
> database, a rc or something else).
Why not just hold this in memory? It has to be sent by the client every time
anyway.
> The needed (mandatory) information that we must be sure to write up to the
> configuration are:
> Client name: a name communicated by the client itself
Each new connection to server "session" has a unique session ID, so no problem
here.
> MIME Types: requested MIME types that the client needs to work properly
Would be send as part of LOGIN command, can be maintained in the session.
> Types of items: requested types of items that the client needs to work
> properly
>
> 3.2 Capability
> The capability is in fact a property of the clients, but is treated here
> because it has to be communicated by the clients to the hypervisor, via
> Dbus.
I disagree. It's a capability of the server. We just need a bit more
information from clients, but otherwise this should be completely hidden from
clients.
All in all pretty good proposal, thanks for it! One note however: we need a
different approach to resources and to agents, see why:
In your proposal you suggest handling agents and resources as one, but that
would not work well: resources are started when there's a client working with
their data, while agents have to be started based on some outside condition
(there are no clients working directly with agents), which makes a big
difference.
Example:
We want Nepomuk Feeder Agent to be running only when Nepomuk is running (and
we don't care what resources are or are not running).
One possible solution how to deal with this is something I call "hooks" (which
is not a good name, but I'm very bad at naming stuff). Hooks would be little
plugins loaded by Akonadi server. They would simply have start(QString
&instance) and stop(QString &instance) signals. So if we use our example
again, hook for Akonadi Nepomuk Feeder Agent would emit
start("akonadi_nepomuk_feeder") signal, when it detects that Nepomuk has been
started and would emit stop("akonadi_nepomuk_feeder") when Nepomuk server
disappears. Based on the signal, Akonadi server would use akonadi_control DBus
interface to have the respective agent started or stopped.
I'm not saying that this solution cannot share some code with your proposed
solution for resources, but the concept needs to be a bit different, since the
concept of agents and resources is a bit different.
Second problem with agents: some agents have a configuration UI used by client
apps - we can use the New Mail Notifier Agent as an example. In KMail, clicking
on a button in accounts settings dialog, a notification configuration dialog is
brought up. The dialog is actually owned by the agent and when you click "OK",
the agent can immediately apply the configuration. If the agent would not be
running then, well, it would not be able to show the configuration :-) Proposed
solution would be to move such configuration into standalone KCM modules, which
would work independently of whether the agent is running. The "hook" for this
agent would just watch the configuration file and when it changes, it would
start (or stop) the agent, depending on what content of the file is - so when
user disables notifications, the hook would emit the stop signal as there's no
need for having notifier agent running if it won't ever send any notification.
Another thing to consider (and that will probably require some larger changes)
is how to tell clients about all resources. Right now if a client wants to
list all available resources and agents, it simply lists all services with
org.freedesktop.Akonadi.Agent.* interface. If we will be shutting down these
resources, they would not appear on DBus and thus could not be listed, so we
need a different way to list them - probably through another extension to
Akonadi protocol or Akonadi's DBus interface that would allow listing all
resources (even those shut down, with a new status like "Suspended" or
something) and a centralized way to configure them - so probably again, instead
of calling org.freedesktop.Akonadi.Agent.Control.configure(), there would have
to be a new method like
org.freedesktop.Akonadi.ResourceManager.configure(QString instance) implemented
no the server, where the Akonadi server would check whether the instance is
running (and start it if necessary) and call it's configure() method.
Ok, I think I've mentioned everything I wanted for now (except for one thing I
can't remember right now, so I'll follow up when I remember).
Also if anyone else made it down here, please share your thoughts and ideas on
this. I think it's a pretty good idea that can help us not to be the "most
resources-hungry" project on embedded devices, and to be less hungry on
desktops too. Judging from general feedback, I think many people will
appreciate that :-)
Cheers,
Dan
[0] http://bit.ly/smgafp
--
Daniel Vrátil
KDE Desktop Team
Associate Software Engineer, Red Hat, Inc.
GPG Key: 0xC59D614F6F4AE348
Fingerprint: 4EC1 86E3 C54E 0B39 5FDD B5FB C59D 614F 6F4A E348
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 490 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.kde.org/pipermail/kde-pim/attachments/20131001/3a591d7f/attachment.sig>
-------------- next part --------------
_______________________________________________
KDE PIM mailing list kde-pim at kde.org
https://mail.kde.org/mailman/listinfo/kde-pim
KDE PIM home page at http://pim.kde.org/
More information about the kde-pim
mailing list