Slow scanning of music directories

[Dan Meltzer]

On Sun, Dec 14, 2008 at 1:01 AM, Big O <illogical1 at gmail.com> wrote:
>
> On Dec 14, 2008, at 12:44 AM, Ian Monroe wrote:
>
>> On Sat, Dec 13, 2008 at 10:29 PM, Big O <illogical1 at gmail.com> wrote:
>>> For a while now I've been sitting on this bug, actually I started
>>> noticing
>>> it ever since we switched over to using mysql-embedded.
>>> I have a 25 Gb ~/Music directory in OS X (there is an additional 10
>>> GB in
>>> /Volumes). The issue is scanning this ~/Music directory became
>>> painfully
>>> slow after switching to mysqle. It seems as if it was even slower
>>> than with
>>> sqlite.
>>> I just moved a directory and the update collection scan took at
>>> least five
>>> minutes to update that 25 GB collection. I say at least five
>>> minutes because
>>> I only noticed after the fan on my laptop got noisy because
>>> Amarok's CPU
>>> usage was > 100% (amarokcollectionscanner itself was using around 16%
>>> consistently).
>>> This means it took > five minutes (7-8 perhaps) to update a
>>> collection
>>> whenever new music lands in the watch path. This is, all kinds of
>>> unacceptable, primarily because of the CPU usage involved; also
>>> because this
>>> isn't even all of my music yet. Does anyone have any idea of what's
>>> going on
>>> and whether or not this is "normal".
>>
>> 7-8 minutes to do a total rescan of 25 gigs is quite fast. Which is
>> probably what is happening.
> A full rescan _seemed_ faster in sqlite. sqlite being dead and all I
> can make hard statements anymore. In linux with mysql server it was
> definitely much faster (Amarok 1.4). If you're making claims to the
> contrary I'd like some hard numbers to back it up.
> Anyone with a 20-30 gig collection fancy doing scan to ease my
> skepticism?

Based on the information your giving (when amarokcollectionscanner
finishes and whats taking the most time) It appears that it's the
actual insertion into the database that is being slow (but I assume
you already knew that).

Try adding some DEBUG_BLOCKS into ScanResultProcessor.cpp to get the
timing of functions and see if you can track down what is taking the
most time.

This might also be aleviated some if (when) we switch to using dbus
for communication between the amarokcollectionscanner and amarok
processes, as then it would be happening simultaneuosly (and be more
multi-threaded friendly)

Dan,
>>
>>
>> So the problem is that incremental scanning isn't working correctly
>> likely.
>>
>> What directory is having a file added to it? Is it a top-level
>> directory?
> The ~/Music directory had a folder added. The folder had one album.
>>
>>
>> Ian
>> _______________________________________________
>> Amarok-devel mailing list
>> Amarok-devel at kde.org
>> https://mail.kde.org/mailman/listinfo/amarok-devel
>
> _______________________________________________
> Amarok-devel mailing list
> Amarok-devel at kde.org
> https://mail.kde.org/mailman/listinfo/amarok-devel
>