Cpp Parser & multibyte chars (bug 274430)

[Andreas Pakulat]

On 19.11.11 10:29:09, Aleix Pol wrote:
> On 11/18/2011 05:55 PM, Milian Wolff wrote:
> >I've spent some time today and investigated bug 274430 [1], which shows that
> >our C++ parser breaks on C-Strings containing wide chars.
> >
> >Andreas tried to convince me in IRC that this is "broken code", since anything
> >besides ASCII in C++ code is undefined.
> >
> >I highly disagree, just because it's undefined doesn't mean one must not use
> >it. Sure, if you are writing portable code one *should* not use it, but at
> >least in my university and probably in science in general, people tend to like
> >utf8 symbols in the output of their computation results. And since most of
> >them are using UTF8 anyways, they will simply put UTF8 chars into their code.
> >
> >So I'd like to fix this, but how? The big issue I see is that our parser
> >operates on QByteArrays (why?) instead of QString, and as such looses all
> >encoding information. Hence our lexer needs two steps to iterate over an "ä"-
> >char instead of one and thusly things it's two chars wide...
> >
> >Any ideas on how to fix this without rewriting the whole parser to use
> >QStrings?
> >
> >[1]: https://bugs.kde.org/show_bug.cgi?id=274430
>
> It's "just" a matter of offsets anyway, so using QString probably
> wouldn't pay off memory-space-wise, but I don't know enough about
> UTF.

In the worst case the memory required by the parser for a single file
would be doubled, since QString uses a 2-byte unicode encoding
internally (i.e. each character is at least 2 bytes). 

But thats not the main problem IMHO, finding out the right encoding is
the crucial point - or writing a function which can translate positions
from the qbytearray into kate's text buffer positions. I'm not sure
which of the two is harder to achieve while keeping the parser working
without ui-dependencies.

Andreas