GSoC update

[Bart Cerneels]

On Fri, Jul 13, 2012 at 11:36 AM, Phalgun Guduthur
<phalgun.guduthur at gmail.com> wrote:
> On Fri, Jul 13, 2012 at 1:02 PM, Bart Cerneels <bart.cerneels at gmail.com>
> wrote:
>>
>> On Wed, Jul 4, 2012 at 4:37 PM, Phalgun Guduthur
>> <phalgun.guduthur at gmail.com> wrote:
>> > Hey All!
>> >
>> > An update on my GSoC project : Semantic Collection for Amarok
>> >
>> > I have figured out a way to reuse a lot of existing Amarok code. So it
>> > made
>> > a part of the code I wrote till now redundant.
>> >
>> > I will be using MemoryCollection as other collections do. The Meta QMaps
>> > will be constructed using queried results from Nepomuk. This will make
>> > all
>> > the collections consistent and hopefully avoids a lot of bugs too.
>> > Similarly, I will also be reusing the MemoryQueryMaker and the other
>> > support
>> > classes that go with it.
>> >
>> > Right now, the Nepomuk Queries are taking too long. So I will have to
>> > run
>> > them in the background using Threadweaver jobs.
>> > Apart from this, the NepomukCollection looks complete with the Meta
>> > classes
>> > implemented.
>>
>> This worries me a little bit. MemoryCollection obviously uses a lot of
>> memory. Where possible we should avoid having to place the complete
>> set of tracks in memory and rather rely on the design goals of
>> QueryMaker to create them on demand. I would think Nepomuk is the
>> perfectly suited for that. Can you explain what blocked you and
>> decided to switch to MemoryCollection?
>>
>> Bart
>
>
> Oh I haven't checked my memory usage when I ran Amarok Collection with
> Nepomuk.
> If all of the MetaMaps are being stored in memory, it should indeed eat up a
> lot, if we consider collections of size more than 2000 tracks. But then,
> wouldn't all the existing collections that use MemoryCollection also suffer
> from the same problem? Say for example the iPodCollection.

They do, but it's not good either. The benefit there is that they
usually don't contain as many tracks as the main collection. Any
implementation that is capable of it should at least attempt to avoid
keeping everything in memory. In your case I think caching of search
results and auto pre-fetching are good strategies.

There is an overuse of MemoryQueryMaker we should try to avoid with
smarter implementations where possible.

For inspiration, a mix of MemoryCollection and on-demand search based
has been implemented by Nikhil Marathe for the 2010 GSoC project
UPnPCollection.

>
> I did speak to Edward (dr_lepper) on the possibility of caching resource
> URI's (These are the unique identifiers of each resource in Nepomuk, just
> strings) of each track so that they can be looked up quickly. But if this
> caching is indeed required, I'll have to postpone it for now and work on
> them after completing other milestones.
>
> And as to why I wanted to reuse the MemoryCollection, I was going to
> implement my NepomukQueryMaker on the same lines as MemoryQueryMaker,
> including the *QueryMakerInternal and other intricacies. Then I realized I
> could make use of MemoryCollection as other collections and maintain the
> same consistency among the collections.
>
> Now I do start to wonder if constructing tracks on demand, say when a user
> clicks on the artist or a genre was indeed an option. Both approaches have
> their pros and cons, 'cos the 'construct at beginning' approach causes a lag
> only during the bootup of Amarok whereas the 'on demand' approach would
> cause a lag every time a query is requested. ( time lag for the query to run
> and enumerate )
>
> Best,
> Phalgun