Slow scanning of music directories

[O]

On Sun, Dec 14, 2008 at 11:54 AM, Dan Meltzer
<parallelgrapefruit at gmail.com> wrote:
> On Sun, Dec 14, 2008 at 1:01 AM, Big O <illogical1 at gmail.com> wrote:
>>
>> On Dec 14, 2008, at 12:44 AM, Ian Monroe wrote:
>>
>>> On Sat, Dec 13, 2008 at 10:29 PM, Big O <illogical1 at gmail.com> wrote:
>>>> For a while now I've been sitting on this bug, actually I started
>>>> noticing
>>>> it ever since we switched over to using mysql-embedded.
>>>> I have a 25 Gb ~/Music directory in OS X (there is an additional 10
>>>> GB in
>>>> /Volumes). The issue is scanning this ~/Music directory became
>>>> painfully
>>>> slow after switching to mysqle. It seems as if it was even slower
>>>> than with
>>>> sqlite.
>>>> I just moved a directory and the update collection scan took at
>>>> least five
>>>> minutes to update that 25 GB collection. I say at least five
>>>> minutes because
>>>> I only noticed after the fan on my laptop got noisy because
>>>> Amarok's CPU
>>>> usage was > 100% (amarokcollectionscanner itself was using around 16%
>>>> consistently).
>>>> This means it took > five minutes (7-8 perhaps) to update a
>>>> collection
>>>> whenever new music lands in the watch path. This is, all kinds of
>>>> unacceptable, primarily because of the CPU usage involved; also
>>>> because this
>>>> isn't even all of my music yet. Does anyone have any idea of what's
>>>> going on
>>>> and whether or not this is "normal".
>>>
>>> 7-8 minutes to do a total rescan of 25 gigs is quite fast. Which is
>>> probably what is happening.
>> A full rescan _seemed_ faster in sqlite. sqlite being dead and all I
>> can make hard statements anymore. In linux with mysql server it was
>> definitely much faster (Amarok 1.4). If you're making claims to the
>> contrary I'd like some hard numbers to back it up.
>> Anyone with a 20-30 gig collection fancy doing scan to ease my
>> skepticism?
>
> Based on the information your giving (when amarokcollectionscanner
> finishes and whats taking the most time) It appears that it's the
> actual insertion into the database that is being slow (but I assume
> you already knew that).
Never occurred to me.
>
> Try adding some DEBUG_BLOCKS into ScanResultProcessor.cpp to get the
> timing of functions and see if you can track down what is taking the
> most time.
Will do. Next week-ish.

>
> This might also be aleviated some if (when) we switch to using dbus
> for communication between the amarokcollectionscanner and amarok
> processes, as then it would be happening simultaneuosly (and be more
> multi-threaded friendly)
>
> Dan,
Thanks for the suggestions, I'll report back with what I found as soon
as possible.

-- 

Bob Hope  - "I don't feel old. I don't feel anything till noon. That's
when it's time for my nap."