[Kde-imaging] Digikam + Kipi + Export and/or Sync to web services (and other such stuff)

Colin Guthrie gmane at colin.guthr.ie
Tue Oct 28 21:09:59 CET 2008


First of all, I'm sorry this comes at such short notice. I had intended 
to get this out over two weeks ago but I've just been swamped at work.

As some of you know I've had an interest in the above topic for some time.

I started off a while ago maintaining the Gallery Export plugin in Kipi 
plugins and subsequently proposed (and started working on) a "sync" 
plugin that was meant to unifiy the various Export plugins currently 
available in kipi-plugins (Flickr export, Picassa export, Gallery export 

I abandoned the sync plugin quite a while back as I was not confident of 
the architecture I had chosen which was separated away from the rest of 
the desktop system and somewhat isolated to kipi which just felt 
"wrong". I felt that there had to be a more appropriate way of 
interacting with these services that would ultimately be more useful in 
the rest of KDE and the desktop.

I considered OpenSync and have noted how systems such as FSpot permit 
sync to Facebook via "conduit" which is built half on top of opensync 
with some python bits chucked in. While I like the opensync approach, it 
was just not in a state to be used when I looked and things are not much 
better now! (/me tries to pretend he has not seen the 0.5 release)

In 2007 I attended Akademy in Glasgow. I sat in a talk on Akonadi and 
although the talk was aimed at PIM data, I immediately thought that this 
could be a very interesting system used for both Amarok and Digikam. Not 
all that long ago, I asked the main developer of the "services" 
framework of Amaork2 about whether he'd looked or was interested in 
Akonadi, but due to the progress already made, he is currently not 
interested in re-engineering it's core to build on top of it 
(unsurprisingly!). While the concept has been mooted on the the 
Digikam/Imaging MLs in the past, I don't think this is particularly 
attractive an option right now to use Akonadi at the core of digikam 
itself. This is all fair enough and I can't doubt the logic behind these 
current decisions and I don't disagree that Akonadi may be overkill for 
accessing a local collection of images. However, I firmly believe that 
the constructs provided by Akonadi are ideally suited to interacting 
with remote systems.

I'd recommend taking a brief wonder around the Akonadi site by way of a 
teaser: http://pim.kde.org/akonadi/

As a brief overview, the Akonadi is built on top of a MySQL server and 
runs as a KDE service. It is essentially a "framework" that represents 
"Collections" and "Items" which maps pretty cleanly to Albums and Images 
in our use case. Akonadi can be able to cache the remote Collections 
meaning that interaction is both quick and responsive. Different 
Collections can have different cache policies attached to them (so for 
example a "View by Tag" collection could be cached less aggressively 
than a "Album" collection). For each "Item" there are three levels of 
granularity that can be affected the caching: Headers, Body and 
Attachments. These are generic and thus can be manipulated for our 
needs. In our case I'd say we'd map this to Metadata, Thumbnail and Full 
Image. Our cache policy will probably be to store only Metadata and 
Thumbnail initially but as the uses for this system expand over time 
this may change (e.g. a screen saver that shows Facebook pictures!!). 
Due to the way that Akonadi works, if part of the data for an item is 
missing (e.g. the Thumbnail or the Full image) it will be fetched from 
the remote system transparently.

This is just a very, very, very brief and hand-wavey description.

So, my (very rough) proposal is approximately as follows:
  0. Discuss the needs of the export and sync systems and decide whether 
Akonadi is a good basis, assuming "yes" proceed.
  1. Read and absorb: 
  2. Take a look over the various web systems with regards to how they 
represent their Collections and Metadata. Map this to a standard 
representation and come up with a set of classes (inspired by Kipi 
representation currently perhaps?)
  3. Write an Akonadi serializer plugin so that our representation can 
be stored in the cache layer. 
  4. Write akonadi "Resources" for each service that implement the 
individual APIs for the given service and map it to our structure. See 
  5. Create a unified "export" plugin in kipi-plugins which will be an 
Akonadi client.

The above is probably somewhat wrong as I don't yet know overly much 
about Akonadi, but I feel this weekend will change that!

The bit I am very unsure about just now is "account" management. I'm not 
sure how you use Akonadi from the perspective of different authenticated 
"accounts". In the PIM context this would be different IMAP accounts, 
but in our context it would be your flickr API key or your facebook 
login etc. Till (for the befit of others, Till is one of the main 
Akonadi developers, also CC'ed) if you have any info on this please feel 
free to add it in :)

So what about Sync I hear you cry? Well, I think that ultimately the 
correct horse to back here is OpenSync and that there will eventually be 
a proper Sync solution that involves a standard synchronisation 
application in KDE (think KitchenSync of old). When this happens, 
integration with Akonadi will certainly be forth coming and 
synchronising your collections and items will be come possible. When 
this happens, I'd propose we create a layer that exposes the Digikam (or 
rather Kipi Host) data via an Akonadi Resource and with this it will be 
possible to "Sync" things via the standard system.

Overall I think using this framework will allow us to create a genuinely 
useful set of interfaces that can be used elsewhere in the desktop and 
also provide a solid foundation for utilising data.

There is definitely (IMO) scope for rebasing the whole KIPI API on top 
of Akonadi but this is perhaps something that is best left for a future 
occasion. Perhaps not, as I know the API breakage is up for discussion, 
but I don't want to bite of more than can be comfortably chewed at the 

By way of fortunate coincidence, the Akonadi team also have a sprint 
planned for this weekend. So if we do decide to take this approach, all 
the main people will be together and available at the same time as 
ourselves. While I'm sure they have their own plans to get on with, I'm 
sure they wont mind taking a little time out to help advise us on this 
plan should we need any assistance!

I have included below my IRC conversation with Till. There is still a 
huge amount I don't know about Akonadi, I just feel that it's an 
appropriate framework to use. I hope other people feel the same :)

See you all on Thursday/Friday :)


Oct 20 21:13:48 *	Now talking on #akonadi
Oct 20 21:13:48 *	Topic for #akonadi is: Akonadi 1.0.0 released: 
http://download.akonadi-project.org/ | general info: 
http://www.akonadi-project.org/ - 
http://techbase.kde.org/Projects/PIM/Akonadi | FAQ: 
http://techbase.kde.org/Projects/PIM/Akonadi#Akonadi_FAQ (includes "Why 
are you using MySQL instead of $DB?" and "Do I need a MySQL server?")
Oct 20 21:13:48 *	Topic for #akonadi set by vkrause at Wed Aug 13 
15:55:47 2008
Oct 20 21:13:55 <coling>	Hello
Oct 20 21:14:13 *	bbroeksema has quit ("Lost terminal")
Oct 20 21:14:22 <coling>	till, are you around just now? I would like to 
ask a question and while I could mail you it would be nice to chat if 
you've got a couple mins?
Oct 20 21:14:35 <till>	Sure.
Oct 20 21:14:40 <coling>	Great :)
Oct 20 21:14:53 <coling>	I saw your talk on Akondi a few years back at 
Akademy in Glasgow
Oct 20 21:15:02 *	bbroeksema 
(n=bbroekse at cust-02-5286b4cb.adsl.scarlet.nl) has joined #akonadi
Oct 20 21:15:23 <coling>	I asked about where you saw the scope *ending* 
for akonadi as it seemed like a good idea and I mentioned perhaps using 
it for e.g. Amarok or Digikam etc.
Oct 20 21:15:53 <coling>	Well, I do some coding for digikam (or rather 
kipi-plugins) and we're going to be having a coding sprint soon in Genoa 
(two weeks time)
Oct 20 21:16:27 <coling>	One of the things we want to do is establish a 
framework for syncronising images from digikam to web-based services 
like facebook, flicker, gallery2 etc.
Oct 20 21:17:16 <coling>	We already have a collection of independant 
plugins that do this job for individial services (e.g. there are 
"flicker export" and "gallery export" etc. plugins.
Oct 20 21:17:39 <coling>	But the problem is the UI is different each 
time and the "account managment" is reimplemented each time too.
Oct 20 21:17:47 <till>	Yep.
Oct 20 21:18:09 <coling>	What I want to do is create a unified system 
for "syncing" (not just export although it could be used for that) with 
these external systems.
Oct 20 21:18:45 <coling>	Where adding a new system is fairly trivial, so 
adding in support for e.g. Picassaweb or whatever would be pretty quick 
and painless.
Oct 20 21:19:27 <coling>	So I've always had akonadi at the back of my 
mind as a framework for doing this.
Oct 20 21:19:57 <coling>	I've also considered trying to incorporate 
opensync somewhere into the mix but as it seems in permanent flux I'm 
not overly confident!
Oct 20 21:20:07 <till>	Hm, sounds a bit like opensync is more what you 
are looking for, to be honest.
Oct 20 21:21:05 <coling>	Well that's kinda what I was wanting to quiz 
you about as I see the latest paper on the akondi website mentioned 
opensync in the title, but I didn't really gather much from the content 
in relation to opensync and how it will work with Akonadi.
Oct 20 21:21:47 <till>	Akonadi is good at caching stuff, providing 
local, fast access to it, keeping track of local changes, etc.
Oct 20 21:22:20 <till>	it has providers/agents/resources to get stuff 
from/to other formats or endpoints.
Oct 20 21:22:26 <till>	So in that case it does some of what you want.
Oct 20 21:22:53 <till>	But then, you already have a local store, you 
don't really need change management, Iguess, nor fine grained caching.
Oct 20 21:23:15 <till>	opensync is about keeping things/files in sync 
between several endpoints.
Oct 20 21:23:29 <till>	where endpoitns can be local apps, mobile devices 
or webservices.
Oct 20 21:23:49 <till>	In that sense Akonadi would be one of many 
opensync endpoints.
Oct 20 21:24:34 <till>	There's some overlap, between the two.
Oct 20 21:24:55 <coling>	Indeed I am beginning to see that :)
Oct 20 21:24:58 <till>	Whether you want akonadi or not kind of depends 
on whether the local store/cache features make sense to you.
Oct 20 21:25:15 <coling>	I certainly like the idea of the local store 
and cache features of your remote accounts.
Oct 20 21:25:46 <coling>	Although I'm not sure if it's a full blown 
absolute must.
Oct 20 21:26:55 <till>	do people want to have several different backends 
with different images, or keep themall in sync?
Oct 20 21:27:21 <coling>	well the idea would be that you could define 
several different backends, e.g. your facebook and your flicker etc.
Oct 20 21:27:32 <coling>	And then do one of two things:
Oct 20 21:27:57 <coling>	1) Manually "Publish" on a per-photo (or 
per-album) basis as a "one-time" operation.
Oct 20 21:28:26 <coling>	2) Define some "sync" settings e.g. publish 
this saved search to facebook as the album "foo"
Oct 20 21:29:08 <coling>	(perhaps not the saved search bit - that's 
probably over complicating thiings for now! but sync this album with 
Oct 20 21:29:51 <coling>	What would be good tho' is the ability to pull 
in comments or tags left by users on these remote systems so you can use 
the metadata locally.
Oct 20 21:32:34 <till>	Hm, I think Akonadi can help quite a bit with that.
Oct 20 21:32:36 <coling>	So I guess what (for me) would be nice is if I 
could define a standard structure for working with this kind of data.
Oct 20 21:32:59 <coling>	You say Akonadi has providers, agents and resources
Oct 20 21:33:08 <till>	Yep.
Oct 20 21:33:17 <coling>	So would I be correct in saying that the 
resources would be the "images"
Oct 20 21:33:40 <coling>	and the build in support for collections would 
represent albums or tags etc.
Oct 20 21:34:09 <coling>	A provider would be a sort of bridging layer 
that joins e.g. facebook to this standard system?
Oct 20 21:34:20 <coling>	And another provider would bridge flickr to it?
Oct 20 21:34:29 <coling>	And then the agents would be what??
Oct 20 21:34:39 <coling>	(or is this all waaaay off base? :))
Oct 20 21:34:39 <till>	No, resources are transports, thigns that feed 
data into the store, and write it back to backends.
Oct 20 21:34:54 <till>	Think kresources, of old.
Oct 20 21:35:22 <till>	agents are processes that operate on the store to 
do things like tagging, manipulation, data mining.
Oct 20 21:36:01 <till>	providers don't exist, I just used that term as a 
analogous one, in case you don't recognize the others. :)
Oct 20 21:36:09 <till>	provides == resources
Oct 20 21:36:13 <coling>	Ahh OK,  So we'd write a resource for each 
separate system we want to use? a Facebook resource and a Flickr 
resource etc.
Oct 20 21:36:15 <coling>	:p
Oct 20 21:36:19 <till>	Yep.
Oct 20 21:36:40 <coling>	And agents would not really be mandatory.
Oct 20 21:36:41 <till>	Each a small process, for robustness reasons.
Oct 20 21:37:08 <till>	No, only if you have things that work with the 
data in the background, o extract info from it, annotate it, etc.
Oct 20 21:37:22 <till>	Your app can do taht too, it would be an agent, 
in that sense.
Oct 20 21:37:31 <till>	Nomenclature is like this:
Oct 20 21:37:40 <till>	clients: all things that read/write the store
Oct 20 21:37:49 <coling>	Cool, I think I've kinda getting a vague handle 
on it thus far!
Oct 20 21:37:53 <till>	agents: non-user facing clients, background 
tasks, etc
Oct 20 21:38:10 <till>	resources: special feeder agents that conect to 
Oct 20 21:38:22 <coling>	OK, I think I'm seeing this working quite 
nicely thus far.
Oct 20 21:38:39 <coling>	It certainly "fits" model wise IMO.
Oct 20 21:38:55 <till>	Yes.
Oct 20 21:39:12 <till>	The nice thing is that the API is fully typed.
Oct 20 21:39:36 <till>	YOu add support for yor mimetype and the C++ type 
for it, and then you can use the API natively with that.
Oct 20 21:39:39 <till>	All templatized.
Oct 20 21:39:57 <coling>	Right so I'd just have to define what I want an 
"image" to represent?
Oct 20 21:40:08 <till>	Yes.
Oct 20 21:40:27 <till>	And write a plugin that can go from/to 
QByteArray, for serialization into the db tables.
Oct 20 21:40:44 <till>	Mostly code for that is already somewhere.
Oct 20 21:40:51 <till>	For images it's easy, Qt already has that code.
Oct 20 21:41:52 <coling>	cool. And would you recommend storing the full 
images or perhaps just a thumb?
Oct 20 21:42:24 <till>	depends on the use acses
Oct 20 21:42:30 <coling>	While storing the full image would be nice, 
should we really be forcing that much data into the db?
Oct 20 21:42:40 <till>	You can have cache policies, we added that for 
online/offline IMAP
Oct 20 21:42:52 <till>	The user can decide, per folder or per resource.
Oct 20 21:43:06 <coling>	Ahh sweet.
Oct 20 21:43:13 <till>	Sometimes they want to keep just headers, 
sometimes full mails, sometimes attachments, sometimes not.
Oct 20 21:43:22 <coling>	So a different policy could be added for e.g. 
your own image vs. those of your friends etc.
Oct 20 21:43:25 <till>	On your phone, you want things lightweight, on 
the desktop, everythign.
Oct 20 21:43:29 <till>	Right.
Oct 20 21:43:40 <till>	What is not there locally is fetched by the 
resource online.
Oct 20 21:43:45 <till>	Transparently.
Oct 20 21:43:50 <coling>	\o/
Oct 20 21:43:53 <coling>	Perfect.
Oct 20 21:44:21 <coling>	OK so the only thing left really is to worry 
about the sync'ing aspect of it.
Oct 20 21:44:36 <till>	The granularity we have is "headers" "body" 
Oct 20 21:44:53 <till>	YOu can map that how you like, maybe to name, 
thumbnail, full image
Oct 20 21:45:17 <till>	For syncing, yes, that goes into the realm of 
opensync, potentially
Oct 20 21:45:24 <coling>	Yeah, comments, tags etc. could be headers I 
Oct 20 21:45:33 <till>	Right.
Oct 20 21:45:36 <coling>	Sounds ideal.
Oct 20 21:45:40 <coling>	I think from the import/publication part of it, 
akonadi sounds perfect.
Oct 20 21:45:54 <coling>	And the syncing part of it perhaps it can go to 
Oct 20 21:45:58 <till>	Could well be.
Oct 20 21:46:03 <coling>	What may be better is to do this:
Oct 20 21:46:17 <coling>	1 Define the resources to work on facebook etc. 
Oct 20 21:46:30 <coling>	2 Define another resource that can connect to 
Oct 20 21:46:44 <coling>	3 Glue together these end points in opensync
Oct 20 21:46:55 <coling>	Would that be possible?
Oct 20 21:46:59 <till>	Yes, for example.
Oct 20 21:47:55 <coling>	So the actual sync part of it is not part of 
the actual stuff we do but we get it for "free" if the templated 
serialization can be exposed to opensync correctly.
Oct 20 21:49:43 <till>	In theory, yes.
Oct 20 21:49:53 <till>	OpenSync as a whole is quite in flux, atm.
Oct 20 21:49:58 <till>	No pun intended.
Oct 20 21:50:17 <coling>	Yeah that's what puts me off using opensync 
directly atm too.
Oct 20 21:51:19 <coling>	I've tried to follow the dev for the last 6 
months or so as a lurker but I haven't got a great feel for the current 
state of affairs and from various discussions with colleagues who fiddle 
with it, I don't think I can realistically develop specifically for it 
right now.
Oct 20 21:51:32 *	bbroeksema has quit ("Lost terminal")
Oct 20 21:52:12 <coling>	But I think doing this work under the structure 
of akonadi, that when opensync support comes up to speed, it should be 
relatively easy to expand things to support it.
Oct 20 21:52:31 <coling>	At least that would be my hope!
Oct 20 21:53:08 <till>	yes, ours as well
Oct 20 21:53:14 <coling>	:)
Oct 20 21:53:20 <till>	we're still hoping our phones will magically 
start working ;)
Oct 20 21:53:28 <coling>	lol
Oct 20 21:53:30 <coling>	So, just to summarise:
Oct 20 21:54:10 <coling>	Do you think this is a fairly legitimate use of 
akonadi? Or do you think I'm trying to squeeze a square peg into a round 
hole by wanting to use it in this scenario?
Oct 20 21:56:07 <till>	No, it seems to fit nicely.
Oct 20 21:56:10 <till>	Worth a try.
Oct 20 22:03:04 <coling>	Cool. OK, thanks for the advice. I'll try and 
draft up a proposal so that we can start work on it. I'll be posting it 
to kde-imaging@ ML and if you don't mind I'll CC you in. You don't have 
to reply but if I say anything too stupid/incorrect perhaps you'd be 
kind enough to gently point that out ;) I think the list is OK to post 
to without subscription so no hassle there, but I'm happy to act as 
go-between if needs be anywa
Oct 20 22:03:05 <coling>	y :)
Oct 20 22:03:38 <coling>	Thanks again for your advice.
Oct 20 22:05:48 <till>	yw


Colin Guthrie

Day Job:
   Tribalogic Limited [http://www.tribalogic.net/]
Open Source:
   Mandriva Linux Contributor [http://www.mandriva.com/]
   PulseAudio Hacker [http://www.pulseaudio.org/]
   Trac Hacker [http://trac.edgewall.org/]

More information about the Kde-imaging mailing list