<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Dawid,</p>
<p>First, I have tried the patch against git head 5.0 branch with no
apparent change in behavior for me with two Merrill Lynch
investment accounts and one Bank of America credit card. I use
PAYEEID for Payee, so I didn't expect any changes. Let me know if
I missed something.</p>
<p>Responses to direct questions inline below.<br>
</p>
<div class="moz-cite-prefix">On 1/24/20 10:45 PM, Dawid Wrobel
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CANO8mVKzhTT9JPu34Ywiqt=L4Xjw81GASokdjHBCtnHN+spSpg@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">Thank you both.
<div><br>
</div>
<div>Jack, correct me if I am wrong, but isn't the
matched Payee replacement behavior you described
typical across the board for all of the plugins?
From what I noticed so far, Payee matching happens
independently in
MyMoneyStatementReader::processTransactionEntry() after
the importing plugin returns a list of
MyMoneyStamtents.</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
I do believe you are correct. My problem is clearly different from
the one you were solving, although at least somewhat related, in
terms of which OFX fields end up in which KMM Transaction fields.<br>
<blockquote type="cite"
cite="mid:CANO8mVKzhTT9JPu34Ywiqt=L4Xjw81GASokdjHBCtnHN+spSpg@mail.gmail.com">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div>However, I do agree that the current matching
is somewhat limiting, since it indeed replaces the
original transaction's payee field. And it is
related to a different limitation I noticed so
far: I wish it was possible to specify Regex rules
for Payee matching using any field, not just the
actual <i>Payee</i>. </div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
I had thought it matched against whichever field you told KMM to use
for Payee.<br>
<blockquote type="cite"
cite="mid:CANO8mVKzhTT9JPu34Ywiqt=L4Xjw81GASokdjHBCtnHN+spSpg@mail.gmail.com">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div>Then, depending on the match, user should be
allowed to decide which of the matches is to be
used for which field. A similar option exists now
in OFX Direct Connect plugin, but it is limited to
Payee and Memo matching only. </div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
Ah - this is what I'm using, so it's why I though the above.
However, I don't think I've noticed any different behavior when
importing an OFX file. Do I need to test better?<br>
<blockquote type="cite"
cite="mid:CANO8mVKzhTT9JPu34Ywiqt=L4Xjw81GASokdjHBCtnHN+spSpg@mail.gmail.com">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div>This is useful and it would be even better if
it was possible to:</div>
<div>
<ul>
<li style="margin-left:15px">make this work for
all importers, by defining these matching
rules per Payee in <i>Payees</i> view, not
separately in plugin's settings.</li>
<li style="margin-left:15px">use any of the
fields, not just <i>Payee</i> and <i>Memo</i>. </li>
<ul>
<li style="margin-left:15px">Specifically,
each of the importing plugins should be
allowed to add an arbitrary list of
key:value fields to MyMoneyStatement object</li>
<li style="margin-left:15px">These could then
be used for a more advanced Payee matching,
e.g.:</li>
<ul>
<li style="margin-left:15px">Match 1: key: <i><font
face="monospace">MEMO</font></i>,
match method: <font face="monospace"><i>Regex</i></font>,
match value:<i> </i><font face="monospace"><i>(<Memo>Online
transfer.*</i>)</font></li>
<li style="margin-left:15px">Match 2: key: <i><font
face="monospace">NAME</font></i>,
match method: <font face="monospace"><i>Regex</i></font>:
match value: <font face="monospace"><i>Payee:
(<Payee>.*)</i></font></li>
<li style="margin-left:15px">...<br>
</li>
<li style="margin-left:15px">Match n: key:<font
face="monospace"><i>ANY</i></font>,
match method: <i><font face="monospace">Exact</font></i>,
match value: <font face="monospace"><i>SOMESPECIFICVALUE</i></font></li>
</ul>
</ul>
<li style="margin-left:15px"><strike>You could
use the named groups (<font face="monospace"><span
style="color:rgb(0,0,0);font-size:medium"><Payee>, </span><span
style="color:rgb(0,0,0);font-size:medium"><Memo>)</span></font>to
force the assignment of the matches to
actual UI fields, <i>or skip assignment
altogether</i> and default to current
behavior</strike> on a second thought, this
is probably too complicated to an average Joe,
so a more explicit UI for assignment could be
used instead.</li>
<ul>
<li style="margin-left:15px"> ideally, it
should be possible to compose Payee and Memo
from more than one match, e.g. <font
face="monospace"><i>Payee: "\1 \2"</i></font>.
Believe it or not, but this is how JPMorgan
CHASE bank's QFX file looks like:</li>
</ul>
</ul>
</div>
</div>
</div>
</div>
</div>
</div>
<blockquote style="margin:0px 0px 0px
40px;border:none;padding:0px">
<blockquote style="margin:0px 0px 0px
40px;border:none;padding:0px"><font face="monospace"><STMTTRN></font></blockquote>
<blockquote style="margin:0px 0px 0px
40px;border:none;padding:0px"><font face="monospace"><TRNTYPE>DEBIT</font></blockquote>
<blockquote style="margin:0px 0px 0px
40px;border:none;padding:0px"><font face="monospace"><DTPOSTED>20200110120000[0:GMT]</font></blockquote>
<blockquote style="margin:0px 0px 0px
40px;border:none;padding:0px"><font face="monospace"><TRNAMT>-5000.00</font></blockquote>
<blockquote style="margin:0px 0px 0px
40px;border:none;padding:0px"><font face="monospace"><FITID>202001100</font></blockquote>
<blockquote style="margin:0px 0px 0px
40px;border:none;padding:0px"><font face="monospace"><NAME><b>Online
Realtime Transfer to Citi</b></font></blockquote>
<blockquote style="margin:0px 0px 0px
40px;border:none;padding:0px"><font face="monospace"><MEMO><b>bank
Checking 1234 transaction#</b></font></blockquote>
<blockquote style="margin:0px 0px 0px
40px;border:none;padding:0px"><font face="monospace"></STMTTRN></font></blockquote>
</blockquote>
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div>
<ul>
<li style="margin-left:15px">it would probably
also be useful to store the original key:value
metadata for future inspection</li>
</ul>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
Ouch. And I though I had issues with Merril Lynch. :-)<br>
<blockquote type="cite"
cite="mid:CANO8mVKzhTT9JPu34Ywiqt=L4Xjw81GASokdjHBCtnHN+spSpg@mail.gmail.com">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div>I do understand this may sound somewhat
complicated, but almost identical functionality
already exists in OFX Direct Connect plugin, so it
would be moved to where I think it makes more
sense and be used by QIF- and CSV-imported
transactions, too.</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
I'll have to leave that to Thomas, but my need still sounds a bit
orthogonal, and is just that if the Payee used for the imported
transaction is found by matching (anything other then being exactly
the same as whatever imported field is used) the keep the original
field as part of the memo, so as not to lose the actual original
payee. (In my case, for example, I have a payee "Misc Restaurants"
and every restaurant I eat at more than once or twice is listed on
the matching tab.<br>
<blockquote type="cite"
cite="mid:CANO8mVKzhTT9JPu34Ywiqt=L4Xjw81GASokdjHBCtnHN+spSpg@mail.gmail.com">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div>From the UI perspective, the way I see it, the
current "List of matching names" option's view
could be modified to make it more detailed –
especially that it already supports regex matching
which one cannot really guess from the UI itself.
While I don't have a mockup handy, I can see how
this can be designed in a clean way. With proper
user manual updates, it would really make
transaction matching a super convenient and
powerful tool. <br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
You'll have to discuss the code with Thomas, but I take care of the
manual, so that's actually easy for me to update at the same time as
the code. (I know there are still outstanding edits to bring the
manual fully up to date with current 5.0.x release.)<br>
<blockquote type="cite"
cite="mid:CANO8mVKzhTT9JPu34Ywiqt=L4Xjw81GASokdjHBCtnHN+spSpg@mail.gmail.com">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div><br>
</div>
<div>PS. Jack, I believe you replied outside of
mailing list by mistake, so I CC-ed kmymoney-devel
again. <br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<p>It was intentional, because I believe it was only a place holder
until I could test. It's fine that the discussion here Is back on
the list. (I also email oddities not worth getting into here, but
I wasn't sure I would be able to reply to the list until I got
home from being away for the weekend. Things worked out for my
laptop, fortunately.)</p>
<p>Jack<br>
</p>
<blockquote type="cite"
cite="mid:CANO8mVKzhTT9JPu34Ywiqt=L4Xjw81GASokdjHBCtnHN+spSpg@mail.gmail.com"><br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Fri, Jan 24, 2020 at 12:41
PM Jack <<a href="mailto:ostroffjh@users.sourceforge.net"
moz-do-not-send="true">ostroffjh@users.sourceforge.net</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">I'll
be glad to check. I might get time over the weekend,
otherwise it <br>
will be early next week.<br>
<br>
By the way - I have also noticed that the Payee field (used
for Payee) <br>
is NOT copied into imported transactions, so if I match many
different <br>
restaurants into the Misc Restaurant payee, I don't know which
on it <br>
actually was. For now, I've been manually adding it back when
I <br>
confirm imported transactions against either the receipt or
the <br>
statement when I reconcile. I'm not sure if that's addressed
in this <br>
patch, but I'll find out soon enough.<br>
<br>
Jack<br>
On 2020.01.24 10:13, Thomas Baumgart wrote:<br>
> Dawid,<br>
> <br>
> thanks for taking the time to improve KMyMoney by
providing a patch. <br>
> The 'messy' code is probably due to historic growth ;)<br>
> <br>
> In fact, the OFX part is not my area of expertise because
none of the <br>
> banks here uses/supports it. But the code looks like we
could use it <br>
> to improve the functionality.<br>
> <br>
> @Jack: can you check if the patch works with your OFX
downloads?<br>
> <br>
> Regards<br>
> <br>
> Thomas<br>
> <br>
> On Freitag, 24. Januar 2020 05:57:25 CET Dawid Wrobel
wrote:<br>
> <br>
> > Hello,<br>
> ><br>
> ><br>
> > The US Citibank's OFX account statement uses NAME
and MEMO fields <br>
> only.<br>
> > Even worse, Payee is saved in MEMO. When importing
using OFX <br>
> plugin, one<br>
> > has an option to choose preferred field for Payee.
However, <br>
> choosing MEMO<br>
> > there solves the problem only partially, since the
NAME field gets<br>
> > discarded despite carrying useful transaction
description <br>
> information that<br>
> > could be otherwise used as Memo.<br>
> ><br>
> ><br>
> > I looked into the code and noticed that it is quite
messy and <br>
> indeed does<br>
> > not consider non-empty NAME field at all, so I
rewrote that section <br>
> of the<br>
> > code.<br>
> ><br>
> ><br>
> > I am building a "matrix" of preference for each of
the Payee, Memo <br>
> and<br>
> > Name, based on the user's choice for Payee. The
importing logic then<br>
> > evaluates each transaction by:<br>
> ><br>
> > - finding the first valid (non-empty) OFX field from
the preferred <br>
> list for<br>
> > Payee;<br>
> ><br>
> > - looking up the corresponding OFX field for Memo
preference. If <br>
> the OFX<br>
> > field is not valid, the corresponding field of the
Name preference <br>
> list is<br>
> > used instead.<br>
> ><br>
> ><br>
> > This opportunistic approach guarantees that no
information gets <br>
> ignored<br>
> > whenever it can be assigned to either Payee or Memo
fields. It also <br>
> handles<br>
> > the investment transactions more cleanly.<br>
> ><br>
> ><br>
> > STEPS TO REPRODUCE<br>
> ><br>
> > 1. Prepare an OFX file with the following
transaction:<br>
> ><br>
> ><br>
> > <STMTTRN><br>
> ><br>
> > <TRNTYPE>CREDIT<br>
> ><br>
> > <DTPOSTED>20200115120000<br>
> ><br>
> > <TRNAMT>1366.31<br>
> ><br>
> > <FITID>202001110001<br>
> ><br>
> > <NAME>ACH Electronic Credit<br>
> ><br>
> > <MEMO>EMPLOYER<br>
> ><br>
> > </STMTTRN><br>
> ><br>
> ><br>
> > 2. Attempt to import, set the preferred Payee field
to MEMO<br>
> ><br>
> ><br>
> > OBSERVED RESULT<br>
> ><br>
> > NAME gets ignored and the MEMO gets used twice
instead<br>
> ><br>
> ><br>
> > EXPECTED RESULT<br>
> ><br>
> > Since MEMO is already used for Payee and PAYEEID is
not provided by <br>
> the<br>
> > OFX, NAME field should be used to populate Memo
information.<br>
> ><br>
> ><br>
> > ADDITIONAL INFORMATION<br>
> ><br>
> > I haven't coded in C++/QT for over 6 years, so my
apologies if the <br>
> code is<br>
> > not up to modern standards.<br>
> ><br>
> ><br>
> > Patch attached and also checked-in to my forked repo
here:<br>
> > <br>
> <a
href="https://github.com/wrobelda/kmymoney/commit/8064ced6da1c4acea582618dbc7777b915124f16"
rel="noreferrer" target="_blank" moz-do-not-send="true">https://github.com/wrobelda/kmymoney/commit/8064ced6da1c4acea582618dbc7777b915124f16</a><br>
> ><br>
> ><br>
> > If required, I can try and push it through
Phabricator instead.<br>
> ><br>
> <br>
> --<br>
> <br>
> Regards<br>
> <br>
> Thomas Baumgart<br>
> <br>
> <a href="https://www.signal.org/" rel="noreferrer"
target="_blank" moz-do-not-send="true">https://www.signal.org/</a>
Signal, the better WhatsApp<br>
>
-------------------------------------------------------------<br>
> Embedded Linux: because you can't do control-alt-delete
on a <br>
> pacemaker.<br>
>
-------------------------------------------------------------<br>
> <br>
<br>
</blockquote>
</div>
</blockquote>
</body>
</html>