Cpp Parser & multibyte chars (bug 274430)

[Aleix Pol]

On 11/19/2011 12:00 PM, Andreas Pakulat wrote:
> On 19.11.11 10:29:09, Aleix Pol wrote:
>> On 11/18/2011 05:55 PM, Milian Wolff wrote:
>>> I've spent some time today and investigated bug 274430 [1], which shows that
>>> our C++ parser breaks on C-Strings containing wide chars.
>>>
>>> Andreas tried to convince me in IRC that this is "broken code", since anything
>>> besides ASCII in C++ code is undefined.
>>>
>>> I highly disagree, just because it's undefined doesn't mean one must not use
>>> it. Sure, if you are writing portable code one *should* not use it, but at
>>> least in my university and probably in science in general, people tend to like
>>> utf8 symbols in the output of their computation results. And since most of
>>> them are using UTF8 anyways, they will simply put UTF8 chars into their code.
>>>
>>> So I'd like to fix this, but how? The big issue I see is that our parser
>>> operates on QByteArrays (why?) instead of QString, and as such looses all
>>> encoding information. Hence our lexer needs two steps to iterate over an "ä"-
>>> char instead of one and thusly things it's two chars wide...
>>>
>>> Any ideas on how to fix this without rewriting the whole parser to use
>>> QStrings?
>>>
>>> [1]: https://bugs.kde.org/show_bug.cgi?id=274430
>> It's "just" a matter of offsets anyway, so using QString probably
>> wouldn't pay off memory-space-wise, but I don't know enough about
>> UTF.
> In the worst case the memory required by the parser for a single file
> would be doubled, since QString uses a 2-byte unicode encoding
> internally (i.e. each character is at least 2 bytes).
>
> But thats not the main problem IMHO, finding out the right encoding is
> the crucial point - or writing a function which can translate positions
> from the qbytearray into kate's text buffer positions. I'm not sure
> which of the two is harder to achieve while keeping the parser working
> without ui-dependencies.
>
> Andreas
Well, a compromise would be to use QString just with those literals that 
require us to. Or maybe just properly calculate the token size, we're 
not storing the literal content anyway, AFAIK.

Regarding encoding, the parser can know what's the encoding before 
starting to parse the file.

Aleix