[Kmymoney-devel] Is the new CSV importer working correctly?

[David Houlden]

On Wednesday 30 November 2011 11:08:35 Allan wrote:
> On 30/11/11 00:46, David Houlden wrote:
> > On Wednesday 30 November 2011 00:28:10 Allan wrote:
> >> On 29/11/11 23:52, Allan wrote:
> >>> On 29/11/11 20:05, Jack wrote:
> >>>> On 2011.11.29 10:35, Allan wrote:
> >>>>> On 29/11/11 15:03, David Houlden wrote:
> >>>>>> I downloaded a csv bank statement from my bank today and ran it
> >>>>>> through the new CSV importer. I found one problem which is maybe a
> >>>>>> little unusual and is down to the way my bank formatted the file. I
> >>>>>> have attached an editted version of the file which contains the
> >>>>>> first record showing the headers and the data record which caused a
> >>>>>> problem.
> >>>>>> 
> >>>>>> The 5th field on the record in question is the Transaction
> >>>>>> Description and it contains a comma. That part of the field is
> >>>>>> however contained within double quote characters. The unusual
> >>>>>> feature is that only part of the field is contained in double
> >>>>>> quotes. The back end (which doesn't contain a comma) is not.
> >>>>>> Importing this file confuses the CSV importer and results in the
> >>>>>> whole of the record from the Transaction Description onwards being
> >>>>>> put into the Transaction Description field even though there are
> >>>>>> other fields delimited by comma.
> >>>>>> 
> >>>>>> I appreciate that this may be unusual data but I would expect that
> >>>>>> each record should be broken into fields using the field delimiter
> >>>>>> but ignoring any field delimiters which are contained within double
> >>>>>> quotes (the text delimiter). Indeed, this file imports as I would
> >>>>>> expect into LibreOffice Calc.
> >>>>>> 
> >>>>>> If required I will raise a bug report for this.
> >>>>> 
> >>>>> I'll have a look at that. I do expect to deal with quotes, but the UI
> >>>>> has undergone a significant upheaval and it may be there's something
> >>>>> I've missed.
> >>>> 
> >>>> I doubt the quotes are the problem; it's likely the fact that there is
> >>>> something after the close quote before the next comma. What library
> >>>> are you using to parse the file, or are you doing it manually? It's
> >>>> probably expecting the next comma immediately after the closing
> >>>> quote, and either getting confused or generating an error at that
> >>>> point.
> >>> 
> >>> I had to write a routine a while ago to deal with the converse problem,
> >>> to detect where a "quoted" string has been erroneously split, because
> >>> of a comma, or, in a value, a 'thousand separator' being mistaken for
> >>> a field delimiter.
> >>> 
> >>> I've now done an addition to deal with this issue and it works.... But,
> >>> as I have to do both checks, the first is conflicting with the new one.
> >>> 
> >>> So, a bit more to do yet. I don't really want to go from scratch, but
> >>> may have to.
> >>> 
> >>> Allan
> >> 
> >> I just needed to make a slight change to the original routine, and now
> >> both conditions are being caught.
> >> 
> >> @Dave
> >> 
> >> I don't know if you'd like to try a patch,before I commit, as you have
> >> the original culprit file?
> > 
> > Sure. Send it and I'll try it out.
> > 
> > Dave.
> 
> I think this should work for you, Dave.
> 
> It's reassuring to know others are trying the plugin, and not just me!
> 
> Allan

Thanks Allan. The patch seems to have fixed the problem I was seeing but has 
introduced another problem. With the patch, quotes around a field or part of a 
field are not removed from the imported data.

Dave