OFX importer should not ignore non-empty Name field if Payee or Memo are empty (patch provided)
Jack
ostroffjh at users.sourceforge.net
Sun Jan 26 22:34:33 GMT 2020
Dawid,
First, I have tried the patch against git head 5.0 branch with no
apparent change in behavior for me with two Merrill Lynch investment
accounts and one Bank of America credit card. I use PAYEEID for Payee,
so I didn't expect any changes. Let me know if I missed something.
Responses to direct questions inline below.
On 1/24/20 10:45 PM, Dawid Wrobel wrote:
> Thank you both.
>
> Jack, correct me if I am wrong, but isn't the matched Payee
> replacement behavior you described typical across the board for all of
> the plugins? From what I noticed so far, Payee matching happens
> independently in
> MyMoneyStatementReader::processTransactionEntry() after the importing
> plugin returns a list of MyMoneyStamtents.
I do believe you are correct. My problem is clearly different from the
one you were solving, although at least somewhat related, in terms of
which OFX fields end up in which KMM Transaction fields.
> However, I do agree that the current matching is somewhat limiting,
> since it indeed replaces the original transaction's payee field. And
> it is related to a different limitation I noticed so far: I wish it
> was possible to specify Regex rules for Payee matching using any
> field, not just the actual /Payee/.
I had thought it matched against whichever field you told KMM to use for
Payee.
> Then, depending on the match, user should be allowed to decide which
> of the matches is to be used for which field. A similar option exists
> now in OFX Direct Connect plugin, but it is limited to Payee and Memo
> matching only.
Ah - this is what I'm using, so it's why I though the above. However, I
don't think I've noticed any different behavior when importing an OFX
file. Do I need to test better?
> This is useful and it would be even better if it was possible to:
>
> * make this work for all importers, by defining these matching rules
> per Payee in /Payees/ view, not separately in plugin's settings.
> * use any of the fields, not just /Payee/ and /Memo/.
> o Specifically, each of the importing plugins should be allowed
> to add an arbitrary list of key:value fields to
> MyMoneyStatement object
> o These could then be used for a more advanced Payee matching, e.g.:
> + Match 1: key: /MEMO/, match method: /Regex/, match
> value:///(<Memo>Online transfer.*/)
> + Match 2: key: /NAME/, match method: /Regex/: match value:
> /Payee: (<Payee>.*)/
> + ...
> + Match n: key:/ANY/, match method: /Exact/, match value:
> /SOMESPECIFICVALUE/
> * You could use the named groups (<Payee>, <Memo>)to force the
> assignment of the matches to actual UI fields, /or skip assignment
> altogether/ and default to current behavior on a second thought,
> this is probably too complicated to an average Joe, so a more
> explicit UI for assignment could be used instead.
> o ideally, it should be possible to compose Payee and Memo from
> more than one match, e.g. /Payee: "\1 \2"/. Believe it or not,
> but this is how JPMorgan CHASE bank's QFX file looks like:
>
> <STMTTRN>
>
> <TRNTYPE>DEBIT
>
> <DTPOSTED>20200110120000[0:GMT]
>
> <TRNAMT>-5000.00
>
> <FITID>202001100
>
> <NAME>*Online Realtime Transfer to Citi*
>
> <MEMO>*bank Checking 1234 transaction#*
>
> </STMTTRN>
>
> * it would probably also be useful to store the original key:value
> metadata for future inspection
>
Ouch. And I though I had issues with Merril Lynch. :-)
> I do understand this may sound somewhat complicated, but almost
> identical functionality already exists in OFX Direct Connect plugin,
> so it would be moved to where I think it makes more sense and be used
> by QIF- and CSV-imported transactions, too.
I'll have to leave that to Thomas, but my need still sounds a bit
orthogonal, and is just that if the Payee used for the imported
transaction is found by matching (anything other then being exactly the
same as whatever imported field is used) the keep the original field as
part of the memo, so as not to lose the actual original payee. (In my
case, for example, I have a payee "Misc Restaurants" and every
restaurant I eat at more than once or twice is listed on the matching tab.
> From the UI perspective, the way I see it, the current "List of
> matching names" option's view could be modified to make it more
> detailed – especially that it already supports regex matching which
> one cannot really guess from the UI itself. While I don't have a
> mockup handy, I can see how this can be designed in a clean way. With
> proper user manual updates, it would really make transaction matching
> a super convenient and powerful tool.
You'll have to discuss the code with Thomas, but I take care of the
manual, so that's actually easy for me to update at the same time as the
code. (I know there are still outstanding edits to bring the manual
fully up to date with current 5.0.x release.)
>
> PS. Jack, I believe you replied outside of mailing list by mistake, so
> I CC-ed kmymoney-devel again.
It was intentional, because I believe it was only a place holder until I
could test. It's fine that the discussion here Is back on the list. (I
also email oddities not worth getting into here, but I wasn't sure I
would be able to reply to the list until I got home from being away for
the weekend. Things worked out for my laptop, fortunately.)
Jack
>
> On Fri, Jan 24, 2020 at 12:41 PM Jack <ostroffjh at users.sourceforge.net
> <mailto:ostroffjh at users.sourceforge.net>> wrote:
>
> I'll be glad to check. I might get time over the weekend,
> otherwise it
> will be early next week.
>
> By the way - I have also noticed that the Payee field (used for
> Payee)
> is NOT copied into imported transactions, so if I match many
> different
> restaurants into the Misc Restaurant payee, I don't know which on it
> actually was. For now, I've been manually adding it back when I
> confirm imported transactions against either the receipt or the
> statement when I reconcile. I'm not sure if that's addressed in this
> patch, but I'll find out soon enough.
>
> Jack
> On 2020.01.24 10:13, Thomas Baumgart wrote:
> > Dawid,
> >
> > thanks for taking the time to improve KMyMoney by providing a
> patch.
> > The 'messy' code is probably due to historic growth ;)
> >
> > In fact, the OFX part is not my area of expertise because none
> of the
> > banks here uses/supports it. But the code looks like we could
> use it
> > to improve the functionality.
> >
> > @Jack: can you check if the patch works with your OFX downloads?
> >
> > Regards
> >
> > Thomas
> >
> > On Freitag, 24. Januar 2020 05:57:25 CET Dawid Wrobel wrote:
> >
> > > Hello,
> > >
> > >
> > > The US Citibank's OFX account statement uses NAME and MEMO fields
> > only.
> > > Even worse, Payee is saved in MEMO. When importing using OFX
> > plugin, one
> > > has an option to choose preferred field for Payee. However,
> > choosing MEMO
> > > there solves the problem only partially, since the NAME field gets
> > > discarded despite carrying useful transaction description
> > information that
> > > could be otherwise used as Memo.
> > >
> > >
> > > I looked into the code and noticed that it is quite messy and
> > indeed does
> > > not consider non-empty NAME field at all, so I rewrote that
> section
> > of the
> > > code.
> > >
> > >
> > > I am building a "matrix" of preference for each of the Payee,
> Memo
> > and
> > > Name, based on the user's choice for Payee. The importing
> logic then
> > > evaluates each transaction by:
> > >
> > > - finding the first valid (non-empty) OFX field from the
> preferred
> > list for
> > > Payee;
> > >
> > > - looking up the corresponding OFX field for Memo preference. If
> > the OFX
> > > field is not valid, the corresponding field of the Name
> preference
> > list is
> > > used instead.
> > >
> > >
> > > This opportunistic approach guarantees that no information gets
> > ignored
> > > whenever it can be assigned to either Payee or Memo fields. It
> also
> > handles
> > > the investment transactions more cleanly.
> > >
> > >
> > > STEPS TO REPRODUCE
> > >
> > > 1. Prepare an OFX file with the following transaction:
> > >
> > >
> > > <STMTTRN>
> > >
> > > <TRNTYPE>CREDIT
> > >
> > > <DTPOSTED>20200115120000
> > >
> > > <TRNAMT>1366.31
> > >
> > > <FITID>202001110001
> > >
> > > <NAME>ACH Electronic Credit
> > >
> > > <MEMO>EMPLOYER
> > >
> > > </STMTTRN>
> > >
> > >
> > > 2. Attempt to import, set the preferred Payee field to MEMO
> > >
> > >
> > > OBSERVED RESULT
> > >
> > > NAME gets ignored and the MEMO gets used twice instead
> > >
> > >
> > > EXPECTED RESULT
> > >
> > > Since MEMO is already used for Payee and PAYEEID is not
> provided by
> > the
> > > OFX, NAME field should be used to populate Memo information.
> > >
> > >
> > > ADDITIONAL INFORMATION
> > >
> > > I haven't coded in C++/QT for over 6 years, so my apologies if
> the
> > code is
> > > not up to modern standards.
> > >
> > >
> > > Patch attached and also checked-in to my forked repo here:
> > >
> >
> https://github.com/wrobelda/kmymoney/commit/8064ced6da1c4acea582618dbc7777b915124f16
> > >
> > >
> > > If required, I can try and push it through Phabricator instead.
> > >
> >
> > --
> >
> > Regards
> >
> > Thomas Baumgart
> >
> > https://www.signal.org/ Signal, the better WhatsApp
> > -------------------------------------------------------------
> > Embedded Linux: because you can't do control-alt-delete on a
> > pacemaker.
> > -------------------------------------------------------------
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.kde.org/pipermail/kmymoney-devel/attachments/20200126/25024596/attachment-0001.html>
More information about the KMyMoney-devel
mailing list