id3, playlists and DB

[Jeff Mitchell]

On Sunday 13 January 2008, Ivan Sergio Borgonovo wrote:
> > I don't quite understand what you mean about not having to change
> > tags and file position, but it's true that UIDs do not take path
>
> Since the only time I changed file positions I also changed path/url
> in the DB I think the problem I'm having can't be related to this
> change.

If you do a full rescan, which you said you've done, it dumps the old info 
anyways.

> > Yeah, you can do that.  Lots of people do.
>
> Server... as running on a centralised server. So that changes to DB
> and fs happens in an atomic way and requests to move files can be
> handled correctly even if they come from several clients at the same
> time.

Well, filesystems are (generally) atomic, and since the database (shouldn't) 
isn't getting updated if the file write fails, this should already be 
possible.

> > mp3tag?  I'm confused now.  Are the issues you were talking about
> > when changing tags in Amarok, or with mp3tag?  Please don't tell me
> > mp3tag comes from id3lib...
>
> mp3tag is an utility I used to check if tags were changed on the
> files. It doesn't work. kid3 works and tells me that it was not an
> issue. Tags are actually written to files too still...

You didn't answer my question of where mp3tag comes from.  But the answer is 
likely to be the same: mp3tag probably only understands up to version 2.3 of 
the ID3 spec.  taglib writes v2.4 tags.  kid3 will be using taglib, so it 
understands 2.4.  But any 2.3 only utility will bail.  I suggest eyeD3 for 
your checking, which reads and writes them all and has some nice verbose 
output if you wish it.

> There are a couple of interface problem: tags display get updated
> too late.
> If you edit a tag "on place", without using the whole "Edit track
> info" form they display the old value. They should update their value
> as soon as you get into "edit mode".

I assume you mean as soon as you get *out* of "edit mode."  If that doesn't 
work for you, then perhaps it's a bug.  I've had some issues with in-place 
tag editing in the past, but I don't do much of it so I really don't remember 
what they were.

> If you use the full form you can actually see the updated tags but if
> you don't change anything the playlist will continue to show the old
> one.

If you don't change anything, there's nothing "old" to show anyways...so this 
isn't a problem.

> > Shouldn't matter for anything.  But it's in the database anyways,
> > so it's shared.
>
> Yep but if I change tags using UIDs as a pk, and the "loaded" is old,
> I'm updating nothing. Just another hypothesis... but that may explain
> why some tags don't get updated when you "edit" on place.

I don't know why you'd do this.  But the UID should never be relied upon for 
anything.  This is by design.  Do not change tags based on UID values in the 
database.  And UIDs will have nothing to do with your editing problems.  As I 
said in my previous emails, UIDs have nothing to do with nothing...

> > I'm fairly certain refreshes happen when you play a file.
> > Otherwise, when the playlist is refetched from the database.  I'm
>
> In my next life I'll understand why some info are in the playlist and
> some others aren't or more precisely why there are duplicated info.

I don't follow.  I'm not sure what info isn't in the playlist that you think 
should be.  Nor do I understand why you expect playlist objects to not have 
any of their own state.  As I've said before, playlist entries can come from 
*many* places, not just the database, so they have to carry their own state 
around with them.

> > not entirely sure why you would expect otherwise.  The playlist is
> > stateful; it doesn't sit there passing data back and forth to the
> > database to see if something's changed.  After all, you can easily
> > have files in your playlist that don't belong in your collection at
> > all (streamed tracks from Ampache, tracks from elsewhere on your
> > filesystem than your collection folders, etc.)
>
> I can understand that a play list may have other sources other than a
> fs... but why duplicating tags?

It's not duplicating tags.  Playlist entries carry meta information of the URL 
that is associated with it.  For files that come from the database, that meta 
information will happen to also reside in the database, where it serves an 
entirely different purpose, i.e. querying.  For files that aren't in the 
collection, that information will be scanned when the file is added to the 
playlist entry, and it will live there and nowhere else.

I understand why you think it's duplicating tags, but you're looking at it in 
a very awkward way.  This is information propagation, not duplication, and it 
has to occur because entries in the playlist cannot rely on being able to 
query the database for their information all the time.  Even those entries 
that *could* do this wouldn't want to, because it'd be miserably slow.

Anyways, that being said, I'm going to skip over the various other places in 
your email where you talk about duplication.  It's not going to change, at 
least not in 1.4.  2.0 probably won't change either, but everything is going 
to shared pointers which will be much more lightweight.

It's simply not feasible to have playlist entries pulled from the collection 
ping back the database to ask if their information has changed from some 
other source.

> > > Updating the collection doesn't have any effect. I'm running
> > > "Rescan" but it is still working. If I get it right it won't help
> > > anyway.
> >
> > No, it won't update the current Playlist on the copy of Amarok
> > where you didn't do the changes.  Nothing will.
>
> Having outdated tag info is a PITA.
> I understand the cost of keeping them constantly in sync, but the
> behaviour I'm seeing still came unexpected in several circumstances.

In two and a half years of hacking Amarok you're the first person I've come 
across that found this unexpected...

...but now that you know better, it is expected behavior for you too.

Honestly, there are generally better ways to use Amarok than have all of your 
songs in the Playlist all the time.  Explore Smart Playlists and Dynamic 
Playlists -- especially Dynamic ones.  Since songs are added not too far in 
advance of when they're played (and you can control this), the information 
should always be relatively "current".

> Consider that listening to music and improving the meta-data aren't
> activities that can be easily separated.
> So it is not just a matter of having another mc or providing an
> interface for editing meta-data.
> Adding meta-data as a "full time job" is boring.

Yes, but this provides an even better case for you using Dynamic Playlists.  
After all, if you're just editing the meta-data of tracks that are being 
played (or being played soon), why do you need them all in the Playlist all 
of the time?

> That could be "on demand" update, and asking the list of all the
> updated tags till yesterday to a DB is going to take less than
> crawling the whole fs or deleting/adding back all the collection to
> the playlist.

The database doesn't store the times that tags are scanned in -- with larger 
collections this would balloon the size of the database even more than it 
already is.  It stores the times that folders are scanned.  These will not 
always be in sync (for instance, if you are editing meta information from the 
Playlist).

> Not that it can make a huge difference... but still...
> pg is on RAID 1, the collection is on a reasonably new PATA HD.
> I do regular maintenance on the DB.
>
> Retrieving from the client all the tags table (~13K rows) takes up to
> 25 sec from psql.

Not too bad, although I've heard pgsql is much slower than mysql.

> Opening amarok and seeing the playlist getting 
> populated takes 70sec.

XML parsing is slow.  Especially when you have 13000 entries.  This is a 
use-case problem you have, not really an Amarok problem.

> Removing everything from the playlist takes 
> 5sec. Repopulating the playlist with all the collection takes up
> 2'30".

Cause you're dragging and dropping...

> > Oh.  Okay.  Use the "All Collection" smart playlist.  When you
> > select all your music from the collection, it has to walk through
> > and build a list of all tracks contained in that entire tree view,
> > whether they're visible to you or not.  It's not a simple SQL
> > query, it's a *lot* of GUI widget processing. There's a good reson
> > Smart Playlists are there, and that "All Collection" is there by
> > default.
>
> ooh oops. That starts to taste sweet. I didn't know you could
> drag&drop the smart play lists. I thought you just could listen to
> them.

It wouldn't really matter, would it?  Either way populates the Playlist with 
the Smart Playlist's tracks.  But I'll bet this takes a lot less time than 
2'30".

> > grumbly things.  I'm also, once again, going to provisionally lay
> > problems at taglib's feet -- if your package is built straight from
> > the 1.4 tarball, you're playing with fire. Most packagers of taglib
> > know this and update with SVN snapshots fairly regularly.  But I
> > don't know about Debian.
>
> Me either. Did you talk with them? I bet they know amarok is a pretty
> popular software and they should be willing to support its need a bit
> more if it is the case.

No, I'm not a Debian user, and as I don't do releases I don't have too much 
contact with packagers.

> > > Another curious behaviour is... you change a tag in a client. It
> > > doesn't show off in the play list. You "Edit track information",
> > > it shows the new one but it doesn't update the playlist.
> >
> > It *should* update unless for some reason it's getting a failed
> > return code from taglib writing the tag.
>
> No... it doesn't feel you "changed" the info it has retrieved from
> the DB and doesn't update what you see in the playlist.
> So even if the "edit track info" form is aware of the change... once
> you close it, it doesn't update the playlist.

Then you're probably hitting a bug somewhere.

> > > As I said I'm not in love with NFS and I'm not in love with Debian
> > > too. But my lan showed in many circumstances to be up to the task
> > > even on comparatively much higher loads on pgsql and nfs.
> > > NFS is not perfect but it was there long before Debian, samba,
> > > kde. It is just a bit younger than NetBIOS.
> >
> > Doesn't have anything to do with loads or throughput -- it has to
> > do with file locking behavior, file access behavior, etc etc etc.
> > NFSv4 may also help.
>
> I'm aware of it, but still other soft can live mostly happy with it.
> Still this shouldn't be the case cos I haven't been so naughty to
> actually access files from both clients.

You'd be surprised how the demands of music tagging and listening can bring 
Samba and NFS to their knees.  :-)

--Jeff