GSoC Proposal: Statistics synchronization for pluggable devices and Last.fm

Wed Mar 21 07:43:47 UTC 2012

On Wed, Mar 21, 2012 at 00:06, Matěj Laitl <matej at laitl.cz> wrote:
> Hi Teo, Bart and list,
> as suggested by Teo, I've decided to try to take part in GSoC 2012 working as
> a student on Amarok. My idea is not on the KDE Ideas page, but I've been
> playing with (a variant of it) for months.
>
> Continues a very draft of my GSoC proposal. I'd be very grateful for any
> possible comments, pointed-out omissions or questions that may arise.
>
> Introduction
> =======
> Amarok has an ability to store per-track play statistics such as play count,
> first & last played date, rating and labels. It then has powerful means to
> generate custom-tailored playlists based on gathered statistics (e.g.: play me
> what I've listened to last month) that many users like to exploit. This works
> well when your computer is the only device you play music from. More likely
> situation is that you play music using Amarok at home, listen to iPod while
> commuting and use Amarok or another music player at work. All these 3 devices
> are able to keep track of what you've listened to, but each one only a third
> of it, which makes Amarok statistics more or less useless. This project aims
> to remedy the situation; Last.fm is an online service that can keep track of
> music a user listens to[1] and can help us with a part of this project. Amarok
> Users group on Last.fm has over 23 000 users.[2]
>
> Project Goals
> ========
> This project will implement:
>  * track statistics synchronization between Amarok collections that support
> statistics; these are currently Local Collection and iPod Collection, but the
> framework will be general
>  * Last.fm scrobbling from pluggable media players that support statistics
> (iPods, currently) using the general framework from previous point
>  * ability to synchronize labels from Amarok to Last.fm
>  * ability to synchronize play counts, first & last played date from Last.fm to
> Amarok collections (other way around is already implemented by scrobbling)
>  * GUI dialogue for performing the synchronization/resolving conflicts
>
> Bonus points (what will Amarok gain for free):
>  * ability to synchronize statistics of Amarok and other media player that
> scrobbles to Last.fm
>  * track statistics backup through Last.fm
>
> Caveats:
>  * Last.fm has no concept of track ratings. This can be however worked around
> by special Last.fm-side labels such as "7/10 stars"
>  * advanced features will be only available for Last.fm users; Last.fm is free
> to use, but the data are public which may be unpleasant for certain users
>
> Implementation
> =========
> Amarok represents audio tracks by Meta::Track abstract C++ class that provides
> getter methods for meta-data (title, artist, album..) and getter/setter
> methods for statistics (rating, play count...). These tracks are grouped into
> so-called Collections, where each Collection represents one source of songs
> (iPod, Local, USB Mass Storage..). Tracks from different collections will be
> matched together using their meta-data and other collection's QueryMaker to
> perform the search. Moreover, iPods provide additional data that can be
> used for conflict-resolution: app_rating and recent_playcount. [3] I plan to
> expose these as new capability offered by Collections. This capability will
> also be used to implement Last.fm scrobbling from iPods (in fact, every
> collection that will support this capability), exploiting recent_playcount
> field in the iPod case. It should be noted that I have already implemented
> similar synchronization in my spare time back in summer 2011 [4], but I was
> not satisfied with its iPod-specific design and GUI, so I decided not to strive
> for its inclusion. But I have the code working and ported to Amarok 2.5, so it
> can be used to fast-start this project.
>
> Another interesting note is that scrobble-from-iPod-to-Last.fm was functional
> in Amarok 1.4 days, but this feature got dropped during rewrites leading to
> 2.0, so this will fix one long-overdue regression.
>
> Speaking about Last.fm integration, Last.fm provides rather nice RESTful API
> [5] a subset of which is already used through liblastfm [6] library in Amarok
> to submit (scrobble) currently played songs. I plan to reuse this library and
> Amarok code dealing with it; the Last.fm API is powerful enough to support all
> claimed features. There is already even Last.fm on-line service Collection,
> but it focuses on playing Last.fm radio streams and doesn't handle individual
> tracks. In order to implement actual synchronization with Last.fm, user's
> Last.fm Library (that contains relevant track data) can be represented as a
> new (invisible) Collection or as special case in synchronizer, I have yet to
> decide this design choice.
>
> Timeline
> =====
> [To be done when the general idea is accepted.] Generally: inter-collection
> synchronization will be first and will be done by the midterm evaluation,
> Last.fm support will be second. I'm already quite bound to Amarok community,
> so I can finish the design, iron-out some details and perhaps present some GUI
> mockups during the community bonding period.
>
> If accepted, GSoC will be my main commitment during the summer, I plan to have
> a week-long vacation and a few 3-day trips, but I'm used to work during
> weekends on open source projects, so the vacation will be compensated.
>
> About me
> ======
> I'm a 24-year-old student of mathematical informatics from Prague, Czech
> Republic. I've been passionate about FLOSS since high school and recently I've
> started contributing to a couple of projects (manly KDE related), most notably
> Amarok where I worked on fixing various bugs singe last autumn [7] and recently
> I've rewritten the iPod collection from scratch [8] as suggested by Amarok's
> Bart Cerneels; I plan to submit a review request for this in coming weeks.
> I've been also particularly active on KDE's bugzilla where I've commented to
> more than 300 bugs.[9] I know C, C++, Python, Java, a bit of French (pun
> intended) and some other less relevant languages. Thanks to my work on Amarok
> I have some experience in GUI programming in Qt & KDE libs.
>
> I've chosen Amarok because I fell in love with it last year, and statistics
> synchronization because it is an area where I found it a bit lacking; I'm a
> music enthusiast who dislikes to listen to the same song twice in a week and I
> believe more Amarok users are that picky and will therefore benefit from this
> work.
>
> I can be reached via e-mail matej at laitl.cz which serves as my Jabber and
> Google account, too. I sometimes hang on #amarok under nick strohel.
>
> [1] http://www.last.fm/user/strohel
> [2] http://www.last.fm/group/Amarok+Users
> [3] http://www.gtkpod.org/libgpod/docs/libgpod-Tracks.html#Itdb-Track
> [4] http://mail.kde.org/pipermail/amarok/2011-June/032736.html
> [5] http://www.last.fm/api/intro
> [6] https://github.com/mxcl/liblastfm/
> [7] https://www.ohloh.net/accounts/strohel
> [8] http://goo.gl/B3Odu
> [9] http://goo.gl/afJN3

I very much see the need for statistics syncing between iPod and SQL
and similar. But I strongly object to a Collection for last.fm, even a
hidden one. It's  corrupting the architecture (Last.fm does not
provide us distinct playable tracks, hence can not be a real
Collection) and further complicates the codebase with various special
cases. We've been suffering a similar mistake with Services already
and want to avoid making it again at all cost.

If all you really need for synchronization from last.fm to a
Collection is metadata and a QueryMaker, I don't see the need for a
Last.fm Collection, QueryMaker, etc. In order to keep it sane you'll
have to execute the sync from the last.fm service anyway and have a
config to select which Collections are synced (SQL as default). From
SQL to last.fm is already implemented with scrobbling.

Very good proposal in any case and I'm confident you can pull it off.

Bart