[Digikam-users] General "DNG and alternatives" discussion regarding long-term archiving.

[Mark Hayes (Hotmail)]

Dear all,

I'm looking for a long-term archiving solution for my RAW files using Linux.  This is not necessarily a Linux problem - I am not yet convinced that the long-term solution for RAW has been found at all yet, on Linux or any other operating system.  And I'd like to start up a discussion about what the current (and forseeable future) solutions might be for me.  I hope you'll add your 2 cent's worth into the discussion.

This is, however, a long e-mail, so you might want to get yourself a coffee before settling down to read it!  :-)

My goal is to have my photos viewable in the distant future by my grandchildren's grandchildren etc etc.  So I'm not just thinking about the next 10 years, my timeline is much longer.  This is more about archiving files for the longer-term future than about having files readable/viewable in the shorter term.

Many people argue that for current RAW files supported by dcraw (or whatever the library is), support and readability will always be there, since the reverse engineering has already been done, the software is Open Sourced and therefore RAW files could always be read in the future.  Whilst this makes sense at first, I am sure that it suffers several problems.

I am not a coder (as will become apparent) and so my arguments may need your 'flexibilty' in understanding them :-)

The FOSS world is an amazing resource of talented individuals and teams that like to write code, wikis, document systems and/or contribute in other ways to the wider community.  Their efforts are tireless and often thankless, and we do not give them enough credit for what they do.  

Nevertheless, they are driven by their desire to solve problems and do cool stuff.  This is often why you see people branch sideways into writing an app for the iPhone to connect to your baseball card collection rather than solve the more mundane areas of getting the printer to work correctly with colour profiles etc.  I don't blame them for it. Who wants to do crap like that?  

However, what it means in the future is that they may easily be more interested in doing cool stuff than fixing the code that makes 100 year old RAW files viewable in the year 2150.  Just because it is POSSIBLE to recode and re-engineer the dcraw code to work in the future on some system of the far-distant future does not mean that someone WILL make that effort.  

It isn't difficult to see that codebases become large over time, and that backwards compatibility for file formats that are long gone will not be top priority.  Someone will then fork it - cut out the crap for files over 100 years old, and maintain the new version for the modern day of the year 2150.  At that point, all RAW files before the cut-off date will be lost and no longer readable.  

This, of course, is not an issue solely about RAW files.  The same argument holds for MS-DOS files, MS Word 6.0 files etc etc.  Over time, there is the very serious potential that file formats that are not in an 'archive' format will not be readable.

Open Source can come to the rescue here - since openly published formats and standards allow for the largest possible ability to reproduce and read a file format.  Hoorah for FOSS!  Nevertheless, again, viewability of any file in the distant future will depend on having files and viewers that are simple to maintain and that can view the widest range of file documents.

Standardisation is key here.  Whilst it doesn't solve all the problems, it does limit the issue of providing the solution to a much smaller field.  Imagine if you only have to write or maintain a viewer for one file format, rather than the abundance of different RAW formats that we currently have.  And imagine if the encoding of the file format was standardised, so that viewing was a much simpler process.  

In addition, imagine having to code photo manipulation software that only has to deal with one type of RAW format.  Wouldn't that make life much easier for all?  And it would allow people to get on with the more cool stuff of providing widgets to manipulate and show off the photos rather than just trying to see the damned things in the first place...

To me, at least, simplification and standardisation, coupled with open standards will give the best possible chance of longer-term survival of my photo files.  This surely has to be the ideal way forward.  

NB.  Please note here that I'm not saying that it is an easy way forward, just the ideal way forward - all companies like to differentiate themselves and give you a reason to buy their products - their version of the RAW format is one way for them to do that.  Getting them to agree to find an archival format (when they haven't yet agreed to DNG) is a large task in itself - but I hope that my arguments for a simplified / singular format might be accepted by many people.

In any case - all of this thinking leads me to believe that for longer-term archival and retrieval of my photos, I need to find an alternative file format that will stand a greater chance of being read in the future.

Anyway, having now convinced myself that an easier and simpler system needs to be found, I find myself looking for one.  Is DNG it?

DNG addresses many aspects that I mention above - but not necessarily all.  And it is certainly one option.  But in terms of being the answer I'm looking for, who knows?  The simple answer is that DNG has been around for a long time now - it's been proposed as a standard, openly published (but not open sourced, is that right?) and put forward for ISO agreement, but nevertheless, it hasn't really made the in-roads into the industry that many would have expected if it was going to be adopted for wider long-term use.  It's not like JPEG, where everyone jumped all over it, is it?

As far as I can see at the moment - my honest opinion is that whilst it's certainly not perfect, DNG is all that there is for now.  Am I missing a glaring other possibility?

Putting aside all arguments about whether Adobe is or isn't the devil incarnate, and whether DNG provides the solution for all aspects of archiving or not, what does a real digital photographic archival solution look like?

Does it look like DNG?

Is there an open alternative?  Should there be?  Should the FOSS community be actively trying to make their own RAW container format (or alternative RAW/Negative format) - that could be used as an alternative?

I don't have answers - which, of course, is why I'm here.  I don't even know if this is the right place to ask this question.  But I hope that others out there may be interested in helping me find a solution...

Once again - I'm not thinking about 10 years from now.  I'm thinking 100 years from now.  Or more.  I'm trying to solve problems now that haven't even happened yet with (many) original file formats since the start of computing.  But that doesn't mean it isn't a problem that needs addressing.  On the contrary, I honestly believe that this is something that the FOSS community should be doing now - and leading the way.  If the answer is not DNG, what does the answer look like?

I'd be grateful for your comments and thoughts on how to long-term archive photographic RAW data for future search and retrieval.  Should I be using DNG as the answer, as a stop-gap until a better solution appears, or what other alternatives are there?

Many thanks,

Mark.

P.S.  You can start flaming me NOW...!!!!