[Digikam-users] database on laptop and server/NAS

[Jean-François Rabasse]

Hello,

Some - late - comments about this very interesting topic.

On Fri, 7 Sep 2012, Manuel Bock wrote:

> The solution used so far was:
> - - - - - - - - - - - - - - - - - - - - - - -
> 
> 1. manage a separate database on the laptop, that writes all tags
> into the images
> 
> 2. work with all the photos taken for example on a holiday trip on
> that laptop
> 
> 3. copy all subdirectories into the server's digikam library tree on
> return and have a digikam instance in my home LAN import them on
> next start up into the server's DB

I have a very similar workflow, main home work done on a home desktop
and extra work, e.g. when in holidays, on a laptop. And, of course,
synchronisation issues once back home.


1. First, I fully agree with what Ben says :

On Mon, 10 Sep 2012, Ben Staude wrote:

> Generally speaking, I think this sort of issues will become more and
> more important for digikam, as images move to NAS devices,
> people use tablets and so on... The setup "one PC, one OS, one HDD"
> is not the typical one for the future. We should plan for an
> (Android|iOS)-app for use with the digikam database...!

Sure, this organisation model will probably reflect less and less
users way of working.
- more and more users have more than one computer, desktops, laptops,
tablets, etc.
- more and more users use non local storage devices, e.g. pluggable
USB drives or NAS.
- probably in a near future, more and more users will use
dematerialised storage, cloud drives et al.

So, rethinking the « one computer - one local drive - one online
collection » model will become necessary in the next few years.
At least, it's a bet.


2. I have also to face another issue : disconnected collection.
This has also been reminded on that thread :

On Mon, 10 Sep 2012, Nick Anderson wrote:

> Disconnected collection browsing would also be useful for this
> usecase.

... and has already been discussed on another thread 
(Off-line Collection Browsing & DB cleanup, 10 to 13 May 2012)

This is a related issue because it's also a consequence of the
current « local drive collection » model. DK requires that all images
be online at any moment as the Scan for new images implementation
assumes that non present images have been removed, instead of being
temporary off-line.

My personal workaround is to keep new images on local drive as long as
I need them for edition tasks, then move new folders and images to an
external USB drive and replace each image file with a reduced size
version.

I agree it's not that great, it's a dirty trick, but as I haven't
enough disk space to hold my collections on a local drive, I still
can use DK to browse and search a reduced size images tree.
And, of course, when I want to access original images after searching,
to be used for a web album or paper prints, I need to plug my USB
storage (on my desktop or my laptop, at convenience).

I do think this off-line collection issue should be considered in a
future, along with support for remote data, NAS, pluggable devices,
cloud devices, etc. Strictly speaking, the physical presence of an
original image file is required (on a local drive) only in two
occasions :
- image edition, of course
- image use, paper print et al.

but for tagging, browsing, searching, only the existence of an image
has to be known, not the effective data. (NB: this is a very old
concept that has been used for centuries by books libraries keepers
and is called « book phantom ». The phantom is a cardboard sheet that
will remain on the shelves (local storage) when a book is borrowed
outside the library (off-line data). That sheet holds the book
informations, title, author, abstract, keywords (metadata) and is
enough for someone to search the library.)


3. Synchronisation issues

I work the same way as Manuel does, edit images metadata and write
into the images files. Synchronisation between computers is done on
the whole tree (that may contain full size new images and reduced size
off-line images). I use tools such as rsync.

This works well, as restarting DK on the target computer will find the
new added images or edited images, and reload metadata from files,
thus updating the database.

This works, but this is manual operations and must be careful
operations. Metadata synchronisation is ambiguous and not errors free,
as Ben says :

On Mon, 10 Sep 2012, Ben Staude wrote:

> With the caveats you always have with such a sync (tagXY assigned on
> laptop, but not on server -> assign tag also on server, or did you
> delete it on server and want it to be deleted on laptop as well?)

Right, and I believe there's no solution today. Having intelligent
synchronisation would require a kind of edition journaling in order to
keep time information along data. Things like « Tag added/modified on
<date-1> » , « Tag removed on <date-2> » , and let a synchronisation
tool rebuild the currently expected metadata state of the document.

As far as I know, there's no such implementation in metadata standards
that describe only data semantics and labels but no historical
information, so the only available solution is : be careful : )


4. Metadata synchronisation issues

On Mon, 10 Sep 2012, Ben Staude wrote:

> For the uniqueHash it might not be a good idea to write metadata
> into the images, but this depends on how this hash is generated.
> Anybody knows?

As appears from the DK source code, the image uniqueHash is a MD5
digest computed from the first and last tenths of kilobytes of the
images. So right, updating metadata in the image file itself,
or editing the image itself, will modify that signature and the
image may be thought as new after some migrations.

It's not a problem - in my opinion - if all metadata is stored into the
image file. DK may think it's a new image, reread metadata, and the DB
will be up to date.
It becomes a problem if extra image related information is stored in
the DB only, then the connection between information and image will be
lost.

My personal choice has been to :
- write metadata to images
- use only metadata writable informations, title, description, tags,
rating, etc. but avoid using features that may leave information in
the database only. This is a major usage restriction.

NB: at some time, I've used the sidecar files mechanism to avoid
writing into the images files. I've stopped using that after having
some disappointments and data losses; it seems (at least with my DK
2.2 version) that the sidecar files implementation is not fully
supported.

E.g. I happened to reorganise folders, search and select some images
sets, then move them via the drag/drop feature into another folder.
DK moves the images correctly, but not the related sidecar files. So,
when copying a collection tree to another computer and running DK on
the target collection, association between images and sidecar files is
broken. I don't know if this has been fixed, perhaps. If not it
probably should be and the « Move image » feature be replaced by
« Move image to target folder and if a sidecar file exists, move it
as well ».


5. Conclusion
I apologise for that long post which is a bit in idle mode; many
comments and questions, no answers nor solutions. But that kind of
topics, multi computers, offline data, appears regularly on this list
and this shows that users working habits evolve with time and are
certainly worth reflexions, discussions, hints sharing...

Regards,
Jean-François