Better database handling of UTF-8 data / Case-insensitive search

[Jeff Mitchell]

Mark Kretschmann wrote:
> Hi Stanislav, big thanks for diagnosing this issue.

Indeed.

Question: what distributions have this problem?

On Gentoo at least, utf8 is the default -- you have to explicitly force
it to use latin1 as the default (compile time option), or explicitly
specify latin1 when creating a table.  I would have thought this would
be the case for most unicode-friendly distributions, i.e. almost any
modern distribution.

This isn't to say we shouldn't try to fix it on our end; however, I
think this is also something we should be asking packagers.

For the #1 fix:

The problem I see here is that for people using problematic mysql
packages, any new table or field that is created will not use the right
encoding.  So this is at best a temporary fix, and one that we have to
then remember to carry forward.  I foresee future bug reports.

As for the #2 fix:

When any database update is performed, a full rescan is run, which drops
pretty much all data with the exception of a few key tables.  I don't
believe the values in those tables will generally have problems with
becoming non-unique after encoding changes.  But you mention other issues.


I think that there's only one right way to do this:

1) Clear the tables that would be wiped clean by a full rescan anyways.
2) Create a new DB, and dump the remaining data over.
3) Create the old one again, with the right encoding by default.
4) Dump the data back over.


I actually would support doing this before 2.1 release, and having
people using SVN test it out.  I'm pretty comfy (compared to most) with
the DB code and think I could get this done.  Otherwise, I think the
entire fix would have to wait for 2.2, which will be who-knows-how-far-away.

--Jeff

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 261 bytes
Desc: OpenPGP digital signature
URL: <http://mail.kde.org/pipermail/amarok/attachments/20090514/7e21b16d/attachment.sig>