Digikam raw files and darktable

Sat Jan 7 14:08:12 GMT 2017

On sábado, 7 de enero de 2017 15:03:38 (CET) Gilles Caulier wrote:
> As i explain before, ALL thumbnails generated by digiKam are stored in a
> dedicated database (sqlite or Mysql).
> 
> The thumbnails are stored using wavelets compression format PGF to optimize
> space. In older DK version (3 for ex), we use FreeDesktop way with PNG
> which explode storage for huge collection and take a while to store
> thumbnails files. Also FreeDesktop recommendations is only limited to
> 256x256, which is not enough for Hdpi screen (digiKam can store 512x512 and
> perhaps more in the future with 8K screen).

Okay. But there is no need to change anything in DK for storing thumbnails. 
The only thing needed is to change the way thumbnails are generated and 
updated for raw files. 

Simply, when DK finds a new raw file, instead of loading the embedded jpg file, 
simply call the raw processor and generate the new file. And when the thumbnail 
is dirty (as I explained before), regenerate the thumbnail.

Isn't that possible currently?

> 
> So, no way to change something in DK to store thumbnails for the moment.
> 
> Gilles Caulier
> 
> 2017-01-07 14:41 GMT+01:00 Juan Jose Casafranca <jjcasmar at gmail.com>:
> > On sábado, 7 de enero de 2017 13:51:57 (CET) Gilles Caulier wrote:
> > > 2017-01-07 12:51 GMT+01:00 Juan Jose Casafranca <jjcasmar at gmail.com>:
> > > > I think you are trying to solve the problem in a very complex way.
> > > > 
> > > > First of all, where does digikam currently saves the thumbnails? If
> > 
> > it's
> > 
> > > > in
> > > > the database, then, the only thing needed is a way to generate the
> > > > processed
> > > > raw as a thumbnail and a flag which specify if the thumbnail is up to
> > > > date.
> > > > 
> > > > This flag es very easy to maintain. When loading the database, the
> > > > thumbnails
> > > > are not up to date. So darktable (or any other raw processor) will be
> > > > called
> > > > to generate the thumbnails. When a raw file is opened for edition
> > 
> > through
> > 
> > > > the
> > > > digikam interface, then also mark the thumbnail is not up to date. If
> > 
> > the
> > 
> > > > user
> > > > opens the file for edition from other place different from the digikam
> > > > interface (opening the raw processor directly for example), then do
> > > > anything.
> > > > I know that thumbnail stored and the processed image will not match in
> > > > this
> > > > case, this should be documented. Enable an option to update manually
> > 
> > the
> > 
> > > > thumbnails for raw files.
> > > > 
> > > > Digikam should have an option to choose which raw processor want the
> > 
> > user
> > 
> > > > to
> > > > user. Default one, darktable, raw therapee, whatever. When a user try
> > 
> > to
> > 
> > > > edits
> > > > a raw file, then the raw processor selected is opened.
> > > > 
> > > > I dont think that we need to store any extra information in the xmp
> > 
> > file.
> > 
> > > > Simply, made different software dont override other software xmp tags
> > 
> > (as
> > 
> > > > I
> > > > said in other post, I think this is common sense).
> > > > 
> > > > Digikam would not treat darktable in a special way. I'm just asking
> > 
> > for an
> > 
> > > > interface to connect digikam with any raw processor. This interface
> > 
> > should
> > 
> > > > only need to provide a way to open the raw processor with the selected
> > > > image
> > > > and a way to get back the image from the raw processor.
> > > > 
> > > > I dont really understand why digikam and raw processors should
> > 
> > communicate
> > 
> > > > through the xmp file.
> > > > 
> > > > So, to summarize:
> > > > User selects darktable for raw editing
> > > 
> > > Already possible in Linux desktop, through OpenWith context menu.
> > > 
> > > > User opens a raw file in digikam -> darktable is opened
> > > 
> > > Already possible in Linux Desktop, with default application set in
> > 
> > desktop,
> > 
> > > that can be handle through a keyboard shortcut. I introduced myself in
> > > 5.0.0. Perhaps this need to be improved, but a good desktop settings in
> > > a
> > > best start.
> > > 
> > > > User edits in darktable and close it -> thumbnail is marked as not up
> > 
> > to
> > 
> > > > date
> > > > Thumbnail is updated calling darktable which returns an image (maybe
> > 
> > with
> > 
> > > > a
> > > > LUA script?)
> > > 
> > > DBUS can be used here. It's easy, but... This will only work under
> > > Linux.
> > > Forget MacOS and Windows, DBUS is a waste of time.
> > 
> > What do you propose for windows and mac? Or just implement this as a linux
> > only feature? (Anyway, windows and mac users already has another raw
> > processing workflow which includes other software, dont they?)
> > 
> > > > The thumbnail is stored in digikam the same way other jpeg or current
> > 
> > raw
> > 
> > > > thumbnails are stored
> > > 
> > > On this point, i'm not agree. It"s a regression and an user wish fixed
> > 
> > few
> > 
> > > year ago...
> > 
> > How do you propose to store raw thumbnails then? I don't really know how
> > digikam stores thumbnails for other types, or even if it stores something
> > or
> > simply generate them online.
> > 
> > > Gilles Caulier
> > > 
> > > > On sábado, 7 de enero de 2017 12:33:43 (CET) Remco Viëtor wrote:
> > > > > On samedi 7 janvier 2017 10:31:27 CET Gilles Caulier wrote:
> > > > > > I follow this thread, and that i can said is : RAW workflow is
> > > > > > terrific
> > > > > > and
> > > > > > very complex. I spare few years to undstand all main subtilities
> > > > > > to
> > > > > > implement a first suitbale support of RAW files in digiKam.
> > > > > > 
> > > > > > Read the contextual responses below for the next...
> > > > > > 
> > > > > > 2017-01-07 10:15 GMT+01:00 Remco Viëtor <remco.vietor at wanadoo.fr>:
> > > > > > > On samedi 7 janvier 2017 03:42:35 CET Juan Jose Casafranca 
wrote:
> > > > > > > > I know that darktable writes not only tags, rating etc, but
> > 
> > also
> > 
> > > > the
> > > > 
> > > > > > > > process history.
> > > > > > > > 
> > > > > > > > That's why I want digikan to call darktable when it founds a
> > 
> > raw
> > 
> > > > file.
> > > > 
> > > > > > > > (...)
> > > > > > > 
> > > > > > > Wouldn't it be easier to get an option in Darktable to write a
> > > > 
> > > > thumbnail
> > > > 
> > > > > > > file
> > > > > > > to disk when leaving darkroom mode or changing file there? And
> > 
> > then
> > 
> > > > add
> > > > 
> > > > > > > a
> > > > > > > link
> > > > > > > to that thumbnail in the XMP file. Basically, you add an API for
> > > > > > > external
> > > > > > > programs to provide a thumbnail image (and afaik, KDE has a
> > > > 
> > > > standardised
> > > > 
> > > > > > > way
> > > > > > > to store thumbnails in a directory).
> > > > > > 
> > > > > > This API is limited even if it's standardized by Open Desktop.
> > 
> > digiKam
> > 
> > > > > > drop
> > > > > > this support since a while for multiple technical reasons, as
> > > > > > using
> > > > > > Wavelets compression to reduce space instead PNG, using a dedicted
> > > > > > database
> > > > > > to handle disconnected removable media, be able to store thumb in
> > > > > > a
> > > > 
> > > > remote
> > > > 
> > > > > > database, support more than 256x256 thumbnails size, etc...
> > > > > > 
> > > > > > So no KDE API here. In all case, we want to reduce to the max KDE
> > > > > > dependencies for the future to be porta le everywhere and reduce
> > > > > > complexity.
> > > > > > 
> > > > > > Note : KDE thumbnials use KIO : we drop this support since 5.0,
> > 
> > which
> > 
> > > > DO
> > > > 
> > > > > > NOT WORK outside Linux, and cannot be bundled in AppImage. No KIO
> > 
> > in
> > 
> > > > > > digiKam for the future...
> > > > > 
> > > > > OK, so that is not a viable option. But a clear way for external
> > > > 
> > > > programs to
> > > > 
> > > > > get information (like thumbnails) back to Digikam would be helpful,
> > 
> > and
> > 
> > > > the
> > > > 
> > > > > sidecar could be a channel for that.
> > > > > 
> > > > > > > And Darktable already generates preview
> > > > > > > images/thumbnails (top left in the darkroom window, and of
> > 
> > course in
> > 
> > > > > > > lighttable mode). Better to reuse those if at all possible.
> > > > > > 
> > > > > > Where are stored this preview image ? In a dedicated place... This
> > > > 
> > > > will be
> > > > 
> > > > > > another puzzle to interface digiKam with that...
> > > > > 
> > > > > Ideally, Digikam shouldn't have to worry about if and where those
> > > > 
> > > > thumbnails
> > > > 
> > > > > are stored internally by Darktable. But the code to *generate* those
> > > > > thumbnails is in place already, what's missing is a way to *export*
> > > > > those
> > > > > thumbnails for use with external programs (i.e. Digikam).
> > > > > 
> > > > > > > It's clear that Digikam's editor will not be able to generate a
> > > > > > > thumbnail
> > > > > > > from
> > > > > > > the original RAW
> > > > > > 
> > > > > > Wrong. digiKam use libraw and extract JPEG preview from RAW file
> > > > > > to
> > > > > > generate thumbnails...
> > > > > > 
> > > > > > > and a Darktable XMP. But starting Darktable for each and
> > > > > > > (...)
> > > > > 
> > > > > Sorry, you cut my phrase at the wrong point...
> > > > > What I wanted to say is that Digikam cannot repeat the processing
> > > > 
> > > > Darktable
> > > > 
> > > > > does, i.e. cannot generate a thumbnail by combining the information
> > 
> > from
> > 
> > > > the
> > > > 
> > > > > RAW file and the XMP file (of course Digikam can generate a
> > > > > thumbnail
> > > > 
> > > > from
> > > > 
> > > > > the RAW file only, it's doing just that all the time...)
> > > > > 
> > > > > > A possible solution is to use IPTC field stored in XMP. There is a
> > > > > > IPTC
> > > > > > preview tag to host JPEG preview. It's limited to 256Kb of data
> > 
> > which
> > 
> > > > is
> > > > 
> > > > > > enough to do the job. Problem : Old IPTC support is, in XMP i
> > > > > > don't
> > > > 
> > > > know,
> > > > 
> > > > > > especially to host binary data in XML.
> > > > > > 
> > > > > > Other problem is to explode the sidecar XMP file size. This is why
> > 
> > we
> > 
> > > > use
> > > > 
> > > > > > a
> > > > > > dedicted database....
> > > > > > 
> > > > > > > Another problem with that would be all the other RAW processors
> > 
> > out
> > 
> > > > > > > there.
> > > > > > > Why
> > > > > > > would Digikam treat Darktable in a special way and ignore all
> > > > > > > the
> > > > > > > others?
> > > > > > 
> > > > > > Remember that we use already a RAW processor : libraw. there are
> > > > 
> > > > plenty of
> > > > 
> > > > > > new feature in this processor not yet used in digiKam. The best
> > 
> > way is
> > 
> > > > to
> > > > 
> > > > > > implment new RAW feature in digiKam, at least to import RAW data
> > > > > > in
> > > > > > editor.
> > > > > 
> > > > > Possible, but not relevant to this subject (IMO): there are many
> > 
> > reasons
> > 
> > > > why
> > > > 
> > > > > someone would use Digikam for DAM, and another program for RAW
> > > > 
> > > > development.
> > > > 
> > > > > The point here is to get a *representation* of the image, as
> > 
> > produced by
> > 
> > > > > that other program, in the Digikam thumbnails view.
> > > > > 
> > > > > So, basically, what I'd aim for is:
> > > > > - define a way to get thumbnails from an external source into
> > 
> > Digikam;
> > 
> > > > > - get the external programs to provide (access to) those thumbnails
> > 
> > in a
> > 
> > > > > well- defined way;
> > > > > - find a way to make sure Digikam won't import unchanged thumbnails
> > > > > a
> > > > 
> > > > second
> > > > 
> > > > > time.
> > > > > 
> > > > > Remco