id3, playlists and DB

[Jeff Mitchell]

On Sunday 13 January 2008, Ivan Sergio Borgonovo wrote:
> On Sun, 13 Jan 2008 13:59:48 -0500
>
> > taglib wouldn't be recommended.  It'd be required.  Amarok doesn't
> > work without taglib, period.  You must have it somehow, somewhere.
> > Might not be in sid.  In which case, knowing Debian (two years
> > behind, full of already-fixed bugs), you may want to compile taglib
> > from SVN.  The 1.4 release is too out-of-date to be considered
> > reliable.
>
> I never had the feeling that sid was out-of-date. In the past I had
> the feeling it was like being on a roller coaster with some updates,
> but generally stuff get fixed in days. And I'm not the sysadmin kind
> of guy, so I'm not versed to fix boxes and I've a low tolerance to
> admin problems. I wouldn't have chosen Debian if it didn't let me
> have a reasonably quiet life.

Sid isn't out of date, but it's totally unstable (in all ways).  if you're not 
the sysadmin kind of guy, what are you doing with Debian?  :-)  Why do you 
think Ubuntu is so popular -- it's Debian, except that it works, with minimal 
fussiness.

> Anyway I've taglib 1.4.

Vanilla?  Or some recent SVN snapshot?  If it's vanilla, then I'm going to 
have to insist that you uninstall that and build taglib SVN.  1.4 vanilla is 
just too out of date and buggy.

> > But it should work just fine for centralized archives.  After all,
> > the information is stored in a shared database; you only need one
> > copy of Amarok to update it.
>
> Unless the rows are locked... well you end up changing the UIDs on one
> client and the tag on the other etc...
> Otherwise I can't understand the behaviour I observe.
> And still you wrote you don't have to change tags and file position.
> I didn't outside amarok and I still observe these problems.
> I could understand them if UIDs where computed even taking into
> account path, but it seems not the case.

I don't quite understand what you mean about not having to change tags and 
file position, but it's true that UIDs do not take path into consideration -- 
although the UID is *associated* with a path.  Regardless, UIDs are not your 
problem here, or at least not right now.  Worry about getting Amarok playing 
nicely with your setup before you worry about AFT working nicely with your 
setup :-)

> I'd be nice to have an amarok server, really use read-only mount and
> let the server take care of the changes on the fs and DB.
> So DB operations could always be coherent with fs operations.

Yeah, you can do that.  Lots of people do.

> Anyway I'd bet on the reliability of the services involved (NFS and
> Postgresql).

Agreed on Postgres.

> I use NFS everyday from sid, lenny and I used it from a forgotten
> version of ubuntu (OK... all anarchists, distributions). The
> Postgresql server is also my development server and it accept pretty
> high loads.
> That doesn't mean that there is no chance that a cosmic ray passed
> through an HD platter turning my server in an evil gnome misbehaving
> just with amarok.

I don't know, are evil GNOMEs misbehaving in there?  :-)

> > > So I can't think nfs is involved when I change the genre of 5
> > > files and 4 get changed and 1 doesn't...
> >
> > It could be.  Who knows.
>
> OK... some more info... mp3tag doesn't work. Uberprgramm kid3 showed
> the changes. But still some files show updates just at the second try
> in the playlist.
> Sorry for misleading you.

mp3tag?  I'm confused now.  Are the issues you were talking about when 
changing tags in Amarok, or with mp3tag?  Please don't tell me mp3tag comes 
from id3lib...

> Hypothesis... not updated UIDs between clients?

Shouldn't matter for anything.  But it's in the database anyways, so it's 
shared.

> > > It could be NFS related (I'll check) but I wouldn't consider it an
> > > NFS fault even if I've the feeling that switching that box to NFS4
> > > may help, but LSI/Dell were so *kind* to kill support for CERT ATA
> > > RAID and I'm stuck with sarge on that server for a while.
> >
> > 3ware FTW.  Dell's built-in cards are always crap.
>
> Next time if I really have to save money I'll go for soft-raid.
> SERIOUS, battery backed-up RAID hardware is going to cost me a kidney
> anyway.

True.  But fairly serious, non-battery-backed-up RAID hardware costs *much* 
less and is good for almost all situations (especially when paired with a 
journaled file system that doesn't do aggressive caching like XFS, i.e. ext3 
or reiser).

> > Of course not.  The playlist isn't refetched from the database when
> > you close and open Amarok -- there'd be no way to have all the
> > customizations you may or may not have made to it.  When you close
> > Amarok, the current playlist is saved -- along with all the
> > metadata information -- to a file.  current.xml IIRC.  When you
> > open Amarok, it's loaded back into the playlist.  So it's behaving
> > exactly as expected.
>
> Then how am I going to obtain a refresh of the info associated with a
> playlist?

I'm fairly certain refreshes happen when you play a file.  Otherwise, when the 
playlist is refetched from the database.  I'm not entirely sure why you would 
expect otherwise.  The playlist is stateful; it doesn't sit there passing 
data back and forth to the database to see if something's changed.  After 
all, you can easily have files in your playlist that don't belong in your 
collection at all (streamed tracks from Ampache, tracks from elsewhere on 
your filesystem than your collection folders, etc.)

(Caveat: AFT will tell the playlist entry if the URL is changed to allow it to 
still be played.  But that's a special case.)

> Updating the collection doesn't have any effect. I'm running "Rescan"
> but it is still working. If I get it right it won't help anyway.

No, it won't update the current Playlist on the copy of Amarok where you 
didn't do the changes.  Nothing will.

> I think you're referring to the order of songs when you say
> "customisation".

That, and things like repeat info, queue info, stop-after-this-track info, 
etc.

> Up to my memory most (all??) the information are already stored in
> the DB, duplicating them... well seems the cause of the limit I'm
> experiencing.

It's not quite like that.  The Playlist is made up of playlist items that have 
all the information necessary to display and play a track.  It's not 
organized or accessed like the database, so it's not a flat out duplication.  
It's made up of stateful objects that are given state by the database on 
creation (if the tracks are pulled from the collection, at least).

> Maybe BMP, Bitrate, SampleRate and Filesize aren't stored in the DB.
> And keeping most of these information in more than one place is going
> to cause no few sync problems.

Nope, those are in the database.  Look at the structure of the tags table, if 
you like.

> On the other side keeping them all in the DB will make updating the
> display much easier. You keep a copy of when you last updated your
> "collection" view and ask to the DB all the songs that were changed
> after that moment in spite of traversing the whole fs.

We already have a list of those files.  That's how a small part of AFT works.  
But AFT currently only updates the URL of the media, which keeps it quick, 
without causing delay.  Start asking it to update more and you get very large 
scalability problems with large playlists (especially memory usage).

Since the track info refreshes on play (I think), then it's not really a 
problem anyways.  Generally.  Except in your case, where you want it to 
refresh earlier.


> > > but deleting and repopulating the playlist is
> > > really time consuming
> >
> > It shouldn't be, especially not in 1.4.8.  How many tracks are we
> > talking about?
>
> Some thousands on the aforementioned CERC ATA RAID 1 through a 2x
> 1Gbit connection, P4 2.4GHz, 512Mb RAM.
> The client is a AMD64x2 2Gb RAM.

For a couple thousand tracks?  That should be pretty fast.  I can load 6,000 
tracks in thirteen seconds or so, cold.  Less than two seconds warm.  No RAID 
here, just a simple 7200RPM SATA drive.  And it clears in about one and a 
half seconds.

> > > and prone to crash/freez
> >
> > Again, it shouldn't be.
>
> Unfortunately while I observed few crashes I saw a lot of freeze and
> even when it doesn't freeze, with VERY large collection it becomes so
> slow it is an annoying operation to populate a playlist.
> eg. you select all the songs from the collection, you drag the mouse
> on the playlist area and well if you're "too fast" it doesn't
> understand you decided to paste them there. You do something else and
> after it finished to see what it was going to move... since the mouse
> is elsewhere it doesn't know anymore where to move them and abort the
> operation (no crash... just don't get what it had to do).

Oh.  Okay.  Use the "All Collection" smart playlist.  When you select all your 
music from the collection, it has to walk through and build a list of all 
tracks contained in that entire tree view, whether they're visible to you or 
not.  It's not a simple SQL query, it's a *lot* of GUI widget processing.  
There's a good reson Smart Playlists are there, and that "All Collection" is 
there by default.

> I observed quite a lot of freeze when I use Music-brainz and when I
> do long sessions of tag editing without closing amarok.

Music brainz is probably their fault.  It's kind of a chumpy library.  I don't 
know too many details about it, I've just heard grumbly things.  I'm also, 
once again, going to provisionally lay problems at taglib's feet -- if your 
package is built straight from the 1.4 tarball, you're playing with fire.  
Most packagers of taglib know this and update with SVN snapshots fairly 
regularly.  But I don't know about Debian.

> > > never changed tags outside amarok... just changed them on
> > > different clients. I'd expect changes show up as soon as a
> > > playlist is reloaded, not after some black magic.
> >
> > If you are changing them within Amarok, then, yes, they should.
>
> But it is not happening. Or at least not as I expect.
> Let me narrow down the problem.
> I saw this behaviour when I'm editing a tag for several files at a
> time and some don't get updated. You see amarok "Writing..." but some
> songs may keep the old tag or take another one.

Okay.  Let me know what you find out.  And see my previous comments about 
taglib  :-)

> Another curious behaviour is... you change a tag in a client. It
> doesn't show off in the play list. You "Edit track information", it
> shows the new one but it doesn't update the playlist.

It *should* update unless for some reason it's getting a failed return code 
from taglib writing the tag.

> > > With this exception no direct action on files/tags has been taken
> > > outside amarok but still I get the above mentioned behaviours.
> > > Consider that eg. changes don't show off no matter which client I
> > > use and in both clients collection was rescanned, playlist killed
> > > and restored etc...
> >
> > It'd be interesting to see how things behaved if you used Samba
> > instead. That's what I use, and I haven't seen any of the problems
> > you describe.
>
> As I said I'm not in love with NFS and I'm not in love with Debian
> too. But my lan showed in many circumstances to be up to the task
> even on comparatively much higher loads on pgsql and nfs.
> NFS is not perfect but it was there long before Debian, samba, kde.
> It is just a bit younger than NetBIOS.

Doesn't have anything to do with loads or throughput -- it has to do with file 
locking behavior, file access behavior, etc etc etc.  NFSv4 may also help.

> > > Please don't take it wrong.
> >
> > Don't worry, we're not.  We tend not to take offense when people
> > have weird, wacky isolated problems that no one else has.  We also
> > tend to think it's an artifact of their setup, not an artifact of
> > our program  :-)
>
> It ain't that weird, maybe uncommon to have thousands songs on NFS.
> But NFS is not the kind of thing I'd define weird.

Its behavior sure can be.  :-)

> thanks anyway for helping me in narrowing down the problem and get
> more conscious about the role of playlists.

Sure.

> I've read that playlist management has taken a different direction in
> amarok 2... but as a matter of interface having an interface place
> where to edit/add tags with a DB back end in a multi-user env is
> still a killer feature for me.

:-)

> Maybe making my use case more explicit will put some light on why...
> You may not like all the music you rip from CDs, there may be several
> people in the same environment that don't share exactly the same
> taste, you may have old music you forgot where it came from or didn't
> rip properly.

Yep.

> Sharing meta-info like genre, rating and score, being able to edit
> genre is a good basis on which build up dynamic lists, so that the
> whole family can listen music.

Indeed.  Although would you want to share that information if people have 
different tastes?  Wouldn't they want to have their own ratings?

--Jeff