[kmymoney4] [Bug 371069] CSV plugin mishandles UTF-16 files

[allan via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=371069

--- Comment #7 from allan <agander93 at gmail.com> ---
[from Thomas]
"
Hi Allan,

you found out yourself: the BOM is not wrong, it's missing. I am sure, you 
stumbled over https://en.wikipedia.org/wiki/UTF-16.

I have not looked at the code of the CSV importer at that point, but you could 
check the beginning of the file (4 bytes) and see if the first two match a BOM 
or you find two 0x00 in those four bytes (where that would probably not work 
for Asian countries as they fill the upper byte with their characters).

Reading the data through a QTextStream allows to setup the encoding/decoding. 
Please take a look at QTextStream::setCodec() and setAutoDetectUnicode(), 
though according to the docs I have, the automatic detection should be the 
default. Maybe, you add another UI selector for the Encoding, in case you 
don't have it. See QTextCodec::availableCodecs() for a list of them. Check 
Kate/Kwrite in the Tools/Encding menu how this may look like.

So much for now. If you don't get the codec stuff going, please tell me where 
to find the relevant source code and I take a look at it.

Thomas"

[My reply]
"
I have /had -

QTextStream inStream(&inFile);
QTextCodec* codec =
QTextCodec::codecForMib(m_codecs.value(m_encodeIndex)->mibEnum());
inStream.setCodec(codec);

QString buf = inStream.readAll();
...
(void CSVWizard::readFile(const QString& fname) line c843)
which I nicked from Qt, I think.

I have encoding selection in the file selector.
...
QPointer<QLabel> label = new QLabel(i18n("Encoding"));
  dialog->layout()->addWidget(label);
  //    Add encoding selection to FileDialog
  QPointer<QComboBox> comboBoxEncode = new QComboBox();
  setCodecList(m_codecs, comboBoxEncode);
  comboBoxEncode->setCurrentIndex(m_encodeIndex);
  connect(comboBoxEncode, SIGNAL(activated(int)), this,
SLOT(encodingChanged(int)));
  dialog->layout()->addWidget(comboBoxEncode);

(bool CSVWizard::getInFileName(QString& inFileName) line c798)

I don't see a setAutoDetectUnicode().

I don't think I had auto-selection, but encoding was by manual selection
from the list of codecs, but UTF-16 seems not to work. (in my code).

Allan"

I'm afraid I'm not able to commit to coding, still.

-- 
You are receiving this mail because:
You are the assignee for the bug.