[Kmymoney-devel] Is the new CSV importer working correctly?

[David Houlden]

On Wednesday 30 November 2011 00:28:10 Allan wrote:
> On 29/11/11 23:52, Allan wrote:
> > On 29/11/11 20:05, Jack wrote:
> >> On 2011.11.29 10:35, Allan wrote:
> >>> On 29/11/11 15:03, David Houlden wrote:
> >>>> I downloaded a csv bank statement from my bank today and ran it
> >>>> through the new CSV importer. I found one problem which is maybe a
> >>>> little unusual and is down to the way my bank formatted the file. I
> >>>> have attached an editted version of the file which contains the first
> >>>> record showing the headers and the data record which caused a problem.
> >>>> 
> >>>> The 5th field on the record in question is the Transaction
> >>>> Description and it contains a comma. That part of the field is
> >>>> however contained within double quote characters. The unusual feature
> >>>> is that only part of the field is contained in double quotes. The
> >>>> back end (which doesn't contain a comma) is not. Importing this file
> >>>> confuses the CSV importer and results in the whole of the record from
> >>>> the Transaction Description onwards being put into the Transaction
> >>>> Description field even though there are other fields delimited by
> >>>> comma.
> >>>> 
> >>>> I appreciate that this may be unusual data but I would expect that
> >>>> each record should be broken into fields using the field delimiter
> >>>> but ignoring any field delimiters which are contained within double
> >>>> quotes (the text delimiter). Indeed, this file imports as I would
> >>>> expect into LibreOffice Calc.
> >>>> 
> >>>> If required I will raise a bug report for this.
> >>> 
> >>> I'll have a look at that. I do expect to deal with quotes, but the UI
> >>> has undergone a significant upheaval and it may be there's something
> >>> I've missed.
> >> 
> >> I doubt the quotes are the problem; it's likely the fact that there is
> >> something after the close quote before the next comma. What library are
> >> you using to parse the file, or are you doing it manually? It's probably
> >> expecting the next comma immediately after the closing quote, and either
> >> getting confused or generating an error at that point.
> > 
> > I had to write a routine a while ago to deal with the converse problem,
> > to detect where a "quoted" string has been erroneously split, because of
> > a comma, or, in a value, a 'thousand separator' being mistaken for a
> > field delimiter.
> > 
> > I've now done an addition to deal with this issue and it works.... But,
> > as I have to do both checks, the first is conflicting with the new one.
> > 
> > So, a bit more to do yet. I don't really want to go from scratch, but
> > may have to.
> > 
> > Allan
> 
> I just needed to make a slight change to the original routine, and now
> both conditions are being caught.
> 
> @Dave
> 
> I don't know if you'd like to try a patch,before I commit, as you have
> the original culprit file?
> 
Sure. Send it and I'll try it out.

Dave.