OFX importer should not ignore non-empty Name field if Payee or Memo are empty (patch provided)

Sat Jan 25 03:45:16 GMT 2020

Thank you both.

Jack, correct me if I am wrong, but isn't the matched Payee replacement
behavior you described typical across the board for all of the plugins?
>From what I noticed so far, Payee matching happens independently in
MyMoneyStatementReader::processTransactionEntry() after the importing
plugin returns a list of MyMoneyStamtents.

However, I do agree that the current matching is somewhat limiting, since
it indeed replaces the original transaction's payee field. And it is
related to a different limitation I noticed so far: I wish it was possible
to specify Regex rules for Payee matching using any field, not just the
actual *Payee*. Then, depending on the match, user should be allowed to
decide which of the matches is to be used for which field. A similar option
exists now in OFX Direct Connect plugin, but it is limited to Payee
and Memo matching only. This is useful and it would be even better if it
was possible to:

   - make this work for all importers, by defining these matching rules per
   Payee in *Payees* view, not separately in plugin's settings.
   - use any of the fields, not just *Payee* and *Memo*.
      - Specifically, each of the importing plugins should be allowed to
      add an arbitrary list of key:value fields to MyMoneyStatement object
      - These could then be used for a more advanced Payee matching, e.g.:
         - Match 1: key: *MEMO*, match method: *Regex*, match value:
*(<Memo>Online
         transfer.**)
         - Match 2: key: *NAME*, match method: *Regex*: match value: *Payee:
         (<Payee>.*)*
         - ...
         - Match n: key:*ANY*, match method: *Exact*, match value:
         *SOMESPECIFICVALUE*
      - You could use the named groups (<Payee>, <Memo>)to force the
   assignment of the matches to actual UI fields, *or skip assignment
   altogether* and default to current behavior  on a second thought, this
   is probably too complicated to an average Joe, so a more explicit UI for
   assignment could be used instead.
      -  ideally, it should be possible to compose Payee and Memo from more
      than one match, e.g. *Payee: "\1 \2"*. Believe it or not, but this is
      how JPMorgan CHASE bank's QFX file looks like:

<STMTTRN>

<TRNTYPE>DEBIT

<DTPOSTED>20200110120000[0:GMT]

<TRNAMT>-5000.00

<FITID>202001100

<NAME>*Online Realtime Transfer to Citi*

<MEMO>*bank Checking  1234 transaction#*

</STMTTRN>

   - it would probably also be useful to store the original key:value
   metadata for future inspection

I do understand this may sound somewhat complicated, but almost identical
functionality already exists in OFX Direct Connect plugin, so it would be
moved to where I think it makes more sense and be used by QIF- and
CSV-imported transactions, too.

>From the UI perspective, the way I see it, the current "List of matching
names" option's view could be modified to make it more detailed –
especially that it already supports regex matching which one cannot really
guess from the UI itself. While I don't have a mockup handy, I can see how
this can be designed in a clean way. With proper user manual updates, it
would really make transaction matching a super convenient and powerful
tool.

PS. Jack, I believe you replied outside of mailing list by mistake, so I
CC-ed kmymoney-devel again.

On Fri, Jan 24, 2020 at 12:41 PM Jack <ostroffjh at users.sourceforge.net>
wrote:

> I'll be glad to check.  I might get time over the weekend, otherwise it
> will be early next week.
>
> By the way -  I have also noticed that the Payee field (used for Payee)
> is NOT copied into imported transactions, so if I match many different
> restaurants into the Misc Restaurant payee, I don't know which on it
> actually was.  For now, I've been manually adding it back when I
> confirm imported transactions against either the receipt or the
> statement when I reconcile.  I'm not sure if that's addressed in this
> patch, but I'll find out soon enough.
>
> Jack
> On 2020.01.24 10:13, Thomas Baumgart wrote:
> > Dawid,
> >
> > thanks for taking the time to improve KMyMoney by providing a patch.
> > The 'messy' code is probably due to historic growth ;)
> >
> > In fact, the OFX part is not my area of expertise because none of the
> > banks here uses/supports it. But the code looks like we could use it
> > to improve the functionality.
> >
> > @Jack: can you check if the patch works with your OFX downloads?
> >
> > Regards
> >
> > Thomas
> >
> > On Freitag, 24. Januar 2020 05:57:25 CET Dawid Wrobel wrote:
> >
> > > Hello,
> > >
> > >
> > > The US Citibank's OFX account statement uses NAME and MEMO fields
> > only.
> > > Even worse, Payee is saved in MEMO. When importing using OFX
> > plugin, one
> > > has an option to choose preferred field for Payee. However,
> > choosing MEMO
> > > there solves the problem only partially, since the NAME field gets
> > > discarded despite carrying useful transaction description
> > information that
> > > could be otherwise used as Memo.
> > >
> > >
> > > I looked into the code and noticed that it is quite messy and
> > indeed does
> > > not consider non-empty NAME field at all, so I rewrote that section
> > of the
> > > code.
> > >
> > >
> > > I am building a "matrix" of preference for each of the Payee, Memo
> > and
> > > Name, based on the user's choice for Payee. The importing logic then
> > > evaluates each transaction by:
> > >
> > > - finding the first valid (non-empty) OFX field from the preferred
> > list for
> > > Payee;
> > >
> > > - looking up the corresponding OFX field for Memo preference. If
> > the OFX
> > > field is not valid, the corresponding field of the Name preference
> > list is
> > > used instead.
> > >
> > >
> > > This opportunistic approach guarantees that no information gets
> > ignored
> > > whenever it can be assigned to either Payee or Memo fields. It also
> > handles
> > > the investment transactions more cleanly.
> > >
> > >
> > > STEPS TO REPRODUCE
> > >
> > > 1. Prepare an OFX file with the following transaction:
> > >
> > >
> > > <STMTTRN>
> > >
> > > <TRNTYPE>CREDIT
> > >
> > > <DTPOSTED>20200115120000
> > >
> > > <TRNAMT>1366.31
> > >
> > > <FITID>202001110001
> > >
> > > <NAME>ACH Electronic Credit
> > >
> > > <MEMO>EMPLOYER
> > >
> > > </STMTTRN>
> > >
> > >
> > > 2. Attempt to import, set the preferred Payee field to MEMO
> > >
> > >
> > > OBSERVED RESULT
> > >
> > > NAME gets ignored and the MEMO gets used twice instead
> > >
> > >
> > > EXPECTED RESULT
> > >
> > > Since MEMO is already used for Payee and PAYEEID is not provided by
> > the
> > > OFX, NAME field should be used to populate Memo information.
> > >
> > >
> > > ADDITIONAL INFORMATION
> > >
> > > I haven't coded in C++/QT for over 6 years, so my apologies if the
> > code is
> > > not up to modern standards.
> > >
> > >
> > > Patch attached and also checked-in to my forked repo here:
> > >
> >
> https://github.com/wrobelda/kmymoney/commit/8064ced6da1c4acea582618dbc7777b915124f16
> > >
> > >
> > > If required, I can try and push it through Phabricator instead.
> > >
> >
> > --
> >
> > Regards
> >
> > Thomas Baumgart
> >
> > https://www.signal.org/       Signal, the better WhatsApp
> > -------------------------------------------------------------
> > Embedded Linux: because you can't do control-alt-delete on a
> > pacemaker.
> > -------------------------------------------------------------
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.kde.org/pipermail/kmymoney-devel/attachments/20200124/ca0a07ba/attachment-0001.html>