id3, playlists and DB

[Ivan Sergio Borgonovo]

On Sun, 13 Jan 2008 19:26:29 -0500
Jeff Mitchell <kde-dev at emailgoeshere.com> wrote:

> Sid isn't out of date, but it's totally unstable (in all ways).  if
> you're not the sysadmin kind of guy, what are you doing with
> Debian?  :-)  Why do you think Ubuntu is so popular -- it's Debian,
> except that it works, with minimal fussiness.

I'm a developer. Stuff I develop generally get in a stable Debian in
production sooner or later.

> > Anyway I've taglib 1.4.

> Vanilla?  Or some recent SVN snapshot?  If it's vanilla, then I'm
> going to have to insist that you uninstall that and build taglib
> SVN.  1.4 vanilla is just too out of date and buggy.

They backported some svn patch. Last patch they ported is dated May
2007.

> I don't quite understand what you mean about not having to change
> tags and file position, but it's true that UIDs do not take path

Since the only time I changed file positions I also changed path/url
in the DB I think the problem I'm having can't be related to this
change.

> > I'd be nice to have an amarok server, really use read-only mount
> > and let the server take care of the changes on the fs and DB.
> > So DB operations could always be coherent with fs operations.

> Yeah, you can do that.  Lots of people do.

Server... as running on a centralised server. So that changes to DB
and fs happens in an atomic way and requests to move files can be
handled correctly even if they come from several clients at the same
time.

> > I use NFS everyday from sid, lenny and I used it from a forgotten
> > version of ubuntu (OK... all anarchists, distributions). The
> > Postgresql server is also my development server and it accept
> > pretty high loads.
> > That doesn't mean that there is no chance that a cosmic ray passed
> > through an HD platter turning my server in an evil gnome
> > misbehaving just with amarok.

> I don't know, are evil GNOMEs misbehaving in there?  :-)

It could be. But being evil gnomes they are hard to catch.

> mp3tag?  I'm confused now.  Are the issues you were talking about
> when changing tags in Amarok, or with mp3tag?  Please don't tell me
> mp3tag comes from id3lib...

mp3tag is an utility I used to check if tags were changed on the
files. It doesn't work. kid3 works and tells me that it was not an
issue. Tags are actually written to files too still...

There are a couple of interface problem: tags display get updated
too late.
If you edit a tag "on place", without using the whole "Edit track
info" form they display the old value. They should update their value
as soon as you get into "edit mode".
If you use the full form you can actually see the updated tags but if
you don't change anything the playlist will continue to show the old
one.

> > Hypothesis... not updated UIDs between clients?

> Shouldn't matter for anything.  But it's in the database anyways,
> so it's shared.

Yep but if I change tags using UIDs as a pk, and the "loaded" is old,
I'm updating nothing. Just another hypothesis... but that may explain
why some tags don't get updated when you "edit" on place.

> True.  But fairly serious, non-battery-backed-up RAID hardware

Fairly serious is not serious and half a kidney is not useful to be
kept ;)

> I'm fairly certain refreshes happen when you play a file.
> Otherwise, when the playlist is refetched from the database.  I'm

In my next life I'll understand why some info are in the playlist and
some others aren't or more precisely why there are duplicated info.

> not entirely sure why you would expect otherwise.  The playlist is
> stateful; it doesn't sit there passing data back and forth to the
> database to see if something's changed.  After all, you can easily
> have files in your playlist that don't belong in your collection at
> all (streamed tracks from Ampache, tracks from elsewhere on your
> filesystem than your collection folders, etc.)

I can understand that a play list may have other sources other than a
fs... but why duplicating tags?

> > Updating the collection doesn't have any effect. I'm running
> > "Rescan" but it is still working. If I get it right it won't help
> > anyway.

> No, it won't update the current Playlist on the copy of Amarok
> where you didn't do the changes.  Nothing will.

Having outdated tag info is a PITA.
I understand the cost of keeping them constantly in sync, but the
behaviour I'm seeing still came unexpected in several circumstances.

> > I think you're referring to the order of songs when you say
> > "customisation".

> That, and things like repeat info, queue info,
> stop-after-this-track info, etc.

What are the problems involved in separating them from the info
already stored in the DB?

Consider that listening to music and improving the meta-data aren't
activities that can be easily separated.
So it is not just a matter of having another mc or providing an
interface for editing meta-data.
Adding meta-data as a "full time job" is boring.

> > Up to my memory most (all??) the information are already stored in
> > the DB, duplicating them... well seems the cause of the limit I'm
> > experiencing.

> It's not quite like that.  The Playlist is made up of playlist
> items that have all the information necessary to display and play a
> track.  It's not organized or accessed like the database, so it's

But still there are duplicated info, keep the information on how to
listen to the music in the playlist and info regarding the music in
the DB.

> Nope, those are in the database.  Look at the structure of the tags
> table, if you like.

So... the playlist could be even slimmer, and most of the things are
already in the DB.

> > On the other side keeping them all in the DB will make updating
> > the display much easier. You keep a copy of when you last updated
> > your "collection" view and ask to the DB all the songs that were
> > changed after that moment in spite of traversing the whole fs.

> We already have a list of those files.  That's how a small part of
> AFT works. But AFT currently only updates the URL of the media,
> which keeps it quick, without causing delay.  Start asking it to
> update more and you get very large scalability problems with large
> playlists (especially memory usage).

That could be "on demand" update, and asking the list of all the
updated tags till yesterday to a DB is going to take less than
crawling the whole fs or deleting/adding back all the collection to
the playlist.

> Since the track info refreshes on play (I think), then it's not
> really a problem anyways.  Generally.  Except in your case, where
> you want it to refresh earlier.

I think that the more meta-data you put into a collection the more
you can enjoy your music since it is easier to search the right music
or to generate auto playlist from those meta-data.

Knowing the current situation of your meta-data is halfway to keep
the quality of your meta-data high and manage it.

> > > It shouldn't be, especially not in 1.4.8.  How many tracks are
> > > we talking about?

> > Some thousands on the aforementioned CERC ATA RAID 1 through a 2x
> > 1Gbit connection, P4 2.4GHz, 512Mb RAM.
> > The client is a AMD64x2 2Gb RAM.

> For a couple thousand tracks?  That should be pretty fast.  I can
> load 6,000 tracks in thirteen seconds or so, cold.  Less than two
> seconds warm.  No RAID here, just a simple 7200RPM SATA drive.  And
> it clears in about one and a half seconds.

Not that it can make a huge difference... but still...
pg is on RAID 1, the collection is on a reasonably new PATA HD.
I do regular maintenance on the DB.

Retrieving from the client all the tags table (~13K rows) takes up to
25 sec from psql. Opening amarok and seeing the playlist getting
populated takes 70sec. Removing everything from the playlist takes
5sec. Repopulating the playlist with all the collection takes up
2'30".

> Oh.  Okay.  Use the "All Collection" smart playlist.  When you
> select all your music from the collection, it has to walk through
> and build a list of all tracks contained in that entire tree view,
> whether they're visible to you or not.  It's not a simple SQL
> query, it's a *lot* of GUI widget processing. There's a good reson
> Smart Playlists are there, and that "All Collection" is there by
> default.

ooh oops. That starts to taste sweet. I didn't know you could
drag&drop the smart play lists. I thought you just could listen to
them.

> Music brainz is probably their fault.  It's kind of a chumpy
> library.  I don't know too many details about it, I've just heard

Could be.

> grumbly things.  I'm also, once again, going to provisionally lay
> problems at taglib's feet -- if your package is built straight from
> the 1.4 tarball, you're playing with fire. Most packagers of taglib
> know this and update with SVN snapshots fairly regularly.  But I
> don't know about Debian.

Me either. Did you talk with them? I bet they know amarok is a pretty
popular software and they should be willing to support its need a bit
more if it is the case.

> > But it is not happening. Or at least not as I expect.
> > Let me narrow down the problem.
> > I saw this behaviour when I'm editing a tag for several files at a
> > time and some don't get updated. You see amarok "Writing..." but
> > some songs may keep the old tag or take another one.

> Okay.  Let me know what you find out.  And see my previous comments
> about taglib  :-)

I'll try to make a deb.

> > Another curious behaviour is... you change a tag in a client. It
> > doesn't show off in the play list. You "Edit track information",
> > it shows the new one but it doesn't update the playlist.

> It *should* update unless for some reason it's getting a failed
> return code from taglib writing the tag.

No... it doesn't feel you "changed" the info it has retrieved from
the DB and doesn't update what you see in the playlist.
So even if the "edit track info" form is aware of the change... once
you close it, it doesn't update the playlist.

> > As I said I'm not in love with NFS and I'm not in love with Debian
> > too. But my lan showed in many circumstances to be up to the task
> > even on comparatively much higher loads on pgsql and nfs.
> > NFS is not perfect but it was there long before Debian, samba,
> > kde. It is just a bit younger than NetBIOS.

> Doesn't have anything to do with loads or throughput -- it has to
> do with file locking behavior, file access behavior, etc etc etc.
> NFSv4 may also help.

I'm aware of it, but still other soft can live mostly happy with it.
Still this shouldn't be the case cos I haven't been so naughty to
actually access files from both clients.

> > Maybe making my use case more explicit will put some light on
> > why... You may not like all the music you rip from CDs, there may
> > be several people in the same environment that don't share
> > exactly the same taste, you may have old music you forgot where
> > it came from or didn't rip properly.
> 
> Yep.
> 
> > Sharing meta-info like genre, rating and score, being able to edit
> > genre is a good basis on which build up dynamic lists, so that the
> > whole family can listen music.

> Indeed.  Although would you want to share that information if
> people have different tastes?  Wouldn't they want to have their own
> ratings?

If they have different tastes they can still have their DB or later
you could split further information.
- How things have to be played (playlist)
- How things have to be "named" (shared DB)
- How things have to be rated (shared or personal rating on DB)

Giving the chance to have a collective rating is nice when you listen
to music with others.

thanks again for sharing ideas

-- 
Ivan Sergio Borgonovo
http://www.webthatworks.it