disappearing data or correupt data file

[Thomas Baumgart]

On Dienstag, 3. Oktober 2023 15:59:07 CEST Jack via KMyMoney-devel wrote:

> On 10/3/23 03:25, Thomas Baumgart wrote:
> > On Dienstag, 3. Oktober 2023 01:37:32 CEST Jack via KMyMoney-devel wrote:
> >> TL;DR at some point in the past day or two (during my monthly
> >> reconciliation of my Merrill Lynch investment accounts) a stray
> >> </KMYMONEY-FILE> ended up in the middle of the <PRICES> section of my
> >> xml data file.  I noticed this because I started getting the
> >> consistency-check message about no price on or before the first use of
> >> a security.  That led to noticing that many prices were missing, and I
> >> ended up actually looking in the xml file.
> > That looks weird. As usual, the first question is: which version of KMM
> > (stable or master)? The logic to write the XML file differs significantly
> > between the two.

> Master, compiled within the past few days.  Although I do on occasion 
> use 5.1, I don't recall doing so in the past few weeks.  It might be 
> interesting if there were some  hint in the file about which version 
> saved it.

Done: https://invent.kde.org/office/kmymoney/-/commit/cbae0103197a5c9486704430ee10746290eecd78

> > A few other questions come to my mind:
> > - do you switch between those versions?
> > - do you have auto saving enabled?

> Only autosave on close if file is changed, 10 versions.

That does not harm.

> >> I'm in process (after making several copies of all saved backups from
> >> the past two days) of splicing the missing <PRICEPAIRS> sections the
> >> the whole <REPORTS> section from the last good file into the current
> >> one (I have no budgets or onlinejobs, so those sections are empty.)
> >> I'd love to know how this might have happened, as well as any things to
> >> look for that this type of data surgery might introduce.

> > I have done this in the past when my ONLINEJOBS were gone. Not that I
> > needed them, but I'd like to keep them around. For me this was caused
> > by a faulty implementation of the XML writer for onlinejobs. That was
> > when I re-wrote the XML logic a few months ago.
> >
> > I don't expect any problems if you proceed your surgery. It could be,
> > that a couple of prices are missing in case you have added some in the
> > meantime. Any modification on the reports would also have to be made
> > again. That's all.

> Good.  That matches what I thought.

> >> -------
> >>
> >> I'm doing my monthly reconciliation of all my Merrill Lynch investment
> >> accounts, and I started getting the "no price for first use date" for
> >> one security.  The equity exists, and there are security accounts for
> >> it in two different investmant accounts.  In both cases, the initial
> >> presence was from an Add Shares transaction (it was a dividend from a
> >> related stock) and was all sold over the past few months.  However,
> >> there are currently NO prices for this equity in my file.  Searching
> >> through explicit backups and my autosave backups, the prices are in all
> >> files prior to 17:15 this evening, but missing from all files after
> >> that time.
> >>
> >> Further looking shows that in the earliest bad file, within the
> >> <PRICES>......</PRICES> section there is clearly a problems:
> >>
> >>       <PRICEPAIR from="E000520" to="USD">
> >>         <PRICE date="2023-09-01" price="5159/50" source="Yahoo Finance"/>
> >>         <PRICE date="2023-09-30" price="10287/100" source="Yahoo
> >> Finance"/>
> >>       </PRICEPAIR>
> >>     </PRICES>
> >> </KMYMONEY-FILE>
> >> <PRICEPAIR from="E000525" to="USD">
> >>      <PRICE date="2022-06-22" price="2367/50" source="User"/>
> >>     <PRICE date="2022-09-11" price="2597/50" source="Yahoo Finance"/>
> >> ....
> >>
> >> Where did that </KMYMONEY-FILE> come from?  I have not been editing the
> >> file manually (at least not today :-).
> > It's not only the </KMYMONEY-FILE> but also the </PRICES> entry. And the
> > <REPORTS> sections is missing completely. Does that file have a REPORTS
> > section later on or is it missing completely?
> 
> What I quoted was not all the way to the end of the file.  I'm not at 
> home right now, but I believe all the files saved after the event 
> included <REPORTS /> and <ONLINEJOBS /> which I read as KMM aborting the 
> reading on hitting </KMYMONEY-FILE> and writing a complete file 
> including empty sections where it had no data.  However, that does not 
> account for the those <PRICEPAIR ...> sections with correct Ennnnn and 
> date but wrong amount.  I need to review the autosave files more 
> carefully to see what is actually after the </KMYMONEY-FILE>
> 
> >> Stranger, the current file at that point is
> >>       <PRICEPAIR from="E000520" to="USD">
> >>         <PRICE date="2022-05-24" price="25080049/380000"
> >> source="Transaction"/>
> >>         <PRICE date="2022-06-03" price="3653/50" source="Yahoo Finance"/>
> >> ... all these PRICEs are correct
> >>         <PRICE date="2023-09-30" price="10287/100" source="Yahoo
> >> Finance"/>
> >>       </PRICEPAIR>
> >>       <PRICEPAIR from="E000525" to="USD">
> >>         <PRICE date="2022-06-22" price="89971/1940" source="User"/>
> >>       </PRICEPAIR>
> >>       <PRICEPAIR from="E000526" to="USD">
> >>         <PRICE date="2022-06-24" price="20697/380" source="User"/>
> >>       </PRICEPAIR>
> >>
> >> So it seems to have kept one pricepair for most but not all the
> >> following equities, and the prices kept are for the first date of the
> >> existing pairs, but the price itself seems totally unrelated.
> >>
> >> Of course since everything past that point is gone, I now have no
> >> custom reports either.
> >>
> >> Do I just chalk this up to haveing a bad day, or perhaps sunspots, or
> >> is there any hope of figuring out what might have ac tually happened.
> >>     <REPORTS/>
> >> One odd thing is that I've got several files with </KMYMONEY-FILE> in
> >> the middle of a <PRICEPAIRS> section, so KMM persisted in writing bad
> >> files.  All the accounts and transactions seem fine, it's just the
> >> prices which are messed up (and the reports missing.)
> >>
> >> Any thoughts or suggestions?
> >>
> >> (The reason I don't want to simply revert to the last known good file
> >> is that I'd have to redo over an hour of importing and reconciliation
> >> work, although I know I may yet need to resort to that.
>
> > The reconciliation data is part of the ACCOUNT elements, so it seems
> > to be in good condition.

> It's not so much the actual reconciliation record but my having cleared 
> all the transaction, after editing some of them, and actually having had 
> to create a few do to having had to remove them manually from the qfx 
> file because of wrong security name.

What I wrote about the ACCOUNT section is also true for the TRANSACTIONS.
Those are also located in front of the PRICEPAIRS.

> > Can you reproduce the problem by using the last good file and repeat what
> > you have done to get to the first faulty one (in case you remember/can find
> > out what that was)?

> I wish I knew what I had done.  I suppose I can try using the last good 
> file and jsut redoing all the reconciliation work since that point.

> > Doesn't KMyMoney complain in case you load a faulty file?

> I don't recall any complaints, but then when I'm doing that monthly 
> reconciliation, I might do over a dozen saves during the process - not 
> only one after finishing each account (separate for investment and 
> brokerage) but often after clearing all of each different activity 
> type.  I'll try again opening a bad file when I get home.

What could be of help is to run xmllint on the file. It will show when
the structure is broken. No need to stop KMyMoney for that. Just run it
after saving in another cli session. Just an idea. One could even automate
that using a tool called inotify :)

-- 

Regards

Thomas Baumgart

-------------------------------------------------------------
We had joy, we had fun, we had Linux on our Sun -- anonymous source
-------------------------------------------------------------
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 868 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.kde.org/pipermail/kmymoney-devel/attachments/20231003/1752e309/attachment-0001.sig>