[Digikam-devel] Collection Scanning of Specific Directories

[Marcel Wiesweg]

> > > I have a large number of pictures spread out over a number of
> > > directories.
> > 
> > 10^6? 10^7?
> 
> About 5.5million
> 

Ok, this explains it. That's a lot indeed.
The complete scan is performed within one database transaction, so the changes 
become public only afterwards, and we need to store change notifications. That 
works, but afterwards, with the current simple solution, notifications are 
sent out one by one instead of collapsing them. This is when you run out of 
memory.
I can try to optimize this situation.

Please note: These are a lot of pictures. It would be interesting where 
digikam is hitting walls here ;-) Some operations require linear memory, e.g. 
if you want to display all of your pictures in the view, the view will keep a 
list of all entries in the collection, and the sort-filter model will also 
need memory to do the sorting.

> > > 2)  Perform some kind of incremental scan so that if it crashes during
> > > the initial scan I could restart and pic up where it left off
> > > (eventually getting past the out of memory problems.)
> > 
> > The solution should be not to get out of memory at all ;-)
> 
> Yes, that would be ideal.  :-)
> 
> Thank you,
> 
> Todd
> 
> Application: digiKam (digikam), signal: Aborted
> [Current thread is 0 (LWP 27729)]
> 
> Thread 2 (Thread 0xb27a9b90 (LWP 27731)):
> [KCrash Handler]
> #5  0xffffe424 in __kernel_vsyscall ()
> #6  0xb5445c41 in raise () from /lib/libc.so.6
> #7  0xb5447428 in abort () from /lib/libc.so.6
> #8  0xb56d08f5 in qt_message_output () from /usr/lib/qt4/libQtCore.so.4
> #9  0xb56d09aa in qFatal () from /usr/lib/qt4/libQtCore.so.4
> #10 0xb56fd8c0 in QListData::realloc () from /usr/lib/qt4/libQtCore.so.4
> #11 0xb56fd9a7 in QListData::append () from /usr/lib/qt4/libQtCore.so.4
> #12 0xb57b9b57 in ?? () from /usr/lib/qt4/libQtCore.so.4
> #13 0x084f30e8 in ?? ()
> #14 0x0000002b in ?? ()
> #15 0xb57b65cc in QCoreApplication::postEvent () from
> /usr/lib/qt4/libQtCore.so.4 #16 0xb57b667a in QCoreApplication::postEvent
> () from /usr/lib/qt4/libQtCore.so.4 #17 0xb57c74d8 in ?? () from
> /usr/lib/qt4/libQtCore.so.4
> #18 0x085a8ce0 in ?? ()
> #19 0x80196010 in ?? ()
> #20 0x0859f690 in ?? ()
> #21 0x0000000b in ?? ()
> #22 0xb57c76db in QMetaObject::activate () from /usr/lib/qt4/libQtCore.so.4
> #23 0xb57c7d44 in QMetaObject::activate () from /usr/lib/qt4/libQtCore.so.4
> #24 0xb6da7980 in Digikam::DatabaseWatch::imageChange (this=0x859f690,
> _t1=@0x85a81d4, _t2=@0x85a81d8, _t3=@0xb27a915c) at
> /root/digikam-svn/graphics/digikam/build/digikam/moc_databasewatch.cpp:193
> #25 0xb6da8152 in Digikam::DatabaseWatch::sendImageChange (this=0x859f690,
> cset=@0xb27a915c) at
> /root/digikam-svn/graphics/digikam/libs/database/databasewatch.cpp:241 #26
> 0xb6daea51 in Digikam::DatabaseBackendPrivate::sendToWatch
> (this=0x85a2058, changeset=@0xb27a915c) at
> /root/digikam-svn/graphics/digikam/libs/database/databasebackend_p.h:59
> #27 0xb6daeb4a in
> Digikam::DatabaseBackendPrivate::ChangesetContainer<Digikam::ImageChangese
> t>::sendOut (this=0x85a20b8) at
> /root/digikam-svn/graphics/digikam/libs/database/databasebackend_p.h:94
> #28 0xb6daec34 in Digikam::DatabaseBackendPrivate::transactionFinished
> (this=0x85a2058) at
> /root/digikam-svn/graphics/digikam/libs/database/databasebackend_p.h:112
> #29 0xb70f4876 in Digikam::DatabaseCoreBackend::commitTransaction
> (this=0x85a05e0) at
> /root/digikam-svn/graphics/digikam/libs/database/databasecorebackend.cpp:1
> 151 #30 0xb6daf53a in ~DatabaseTransaction (this=0xb27a9274) at
> /root/digikam-svn/graphics/digikam/libs/database/databasetransaction.cpp:5
> 7 #31 0xb6d932c6 in Digikam::CollectionScanner::completeScan
> (this=0xb27a92f8) at
> /root/digikam-svn/graphics/digikam/libs/database/collectionscanner.cpp:277
> #32 0x08335a0f in Digikam::ScanController::run (this=0x8588f60) at
> /root/digikam-svn/graphics/digikam/digikam/scancontroller.cpp:541 #33
> 0xb56d7fc1 in ?? () from /usr/lib/qt4/libQtCore.so.4
> #34 0x08588f60 in ?? ()
> #35 0x00000000 in ?? ()
> 
> Thread 1 (Thread 0xb3bae700 (LWP 27729)):
> #0  0xffffe424 in __kernel_vsyscall ()
> #1  0xb567e8d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/libpthread.so.0 #2  0xb56d739a in ?? () from
> /usr/lib/qt4/libQtCore.so.4
> #3  0x0859fd80 in ?? ()
> #4  0x0859fd68 in ?? ()
> #5  0x00000000 in ?? ()
> _______________________________________________
> Digikam-devel mailing list
> Digikam-devel at kde.org
> https://mail.kde.org/mailman/listinfo/digikam-devel