[Digikam-users] backup and data integrity

[Arnd Baecker]

Hi,

On Wed, 16 Jan 2008, Jakob [iso-8859-1] Østergaard Hegelund wrote:
> On Wednesday 16 January 2008, Arnd Baecker wrote:
[...]

> > Another (maybe often over-looked?) aspect is whether
> > the data (both on the master disk and the backups)
> > are still correct.
> > It may well happen that files just get corrupted
> > on the hard-disk.
>
> Sure, that's unlikely but not impossible.

Well, that's what I originally thought as well -
all this was triggered by a thread on the pano-tools list,
  http://www.nabble.com/OT:-bad-media-and-old-stuff-(branch-from:-Back-to-film)-td14722818.html
and there people assert that the probability is very high
(and after I had such an experience, it actually
seems more likely than unlikely to me ...).
Getting hard figures seems difficult though:

Maybe the most extensive test is
"Failure Trends in a Large Disk Drive Population"
http://research.google.com/archive/disk_failures.pdf

Another remarkable one is the CERN Data Integrity paper,
http://indico.cern.ch/getFile.py/access?contribId=3&sessionId=0&resId=1&materialId=paper&confId=13797
which I found via this article
http://blogs.zdnet.com/storage/?p=191
(interesting links/discussions there as well)

Also of interest might be:
http://www.adrc.com/diy_rescue/df/data_corruption.html
http://www.naturescapes.net/phpBB2/viewtopic.php?t=95860

[...]

ZFS sounds interesting - thanks for the pointer!
For data integrity on the hard-disk
this looks like a really good solution!
Unfortunately, I would suspect that this would take a while
until this is implemented/fully integrated.
There is a ZFS via FUSE project
  http://developer.berlios.de/projects/zfs-fuse/
see also
  http://kerneltrap.org/node/8066
for a background on the problems with ZFS with respect
to license, patents, GPL, ...

So after reading through the above, to me ZFS unfortunately
does not seem like a near-future solution...

Also, in the long run, windows will be one of the OS under
which digikam can be run (not that I would like to do that ;-),
so maybe a digikam specific solution might be better than
none at all...

Still, how would ZFS allow to check the
integrity of backuped files in comparison with those
on the master hard-disk?

Thanks a lot - keep your ideas coming ;-)

Best, Arnd