disappearing data or correupt data file

[Thomas Baumgart]

On Dienstag, 3. Oktober 2023 01:37:32 CEST Jack via KMyMoney-devel wrote:

> TL;DR at some point in the past day or two (during my monthly  
> reconciliation of my Merrill Lynch investment accounts) a stray  
> </KMYMONEY-FILE> ended up in the middle of the <PRICES> section of my  
> xml data file.  I noticed this because I started getting the  
> consistency-check message about no price on or before the first use of  
> a security.  That led to noticing that many prices were missing, and I  
> ended up actually looking in the xml file.

That looks weird. As usual, the first question is: which version of KMM
(stable or master)? The logic to write the XML file differs significantly
between the two.

A few other questions come to my mind:
- do you switch between those versions?
- do you have auto saving enabled?

> I'm in process (after making several copies of all saved backups from  
> the past two days) of splicing the missing <PRICEPAIRS> sections the  
> the whole <REPORTS> section from the last good file into the current  
> one (I have no budgets or onlinejobs, so those sections are empty.)   
> I'd love to know how this might have happened, as well as any things to  
> look for that this type of data surgery might introduce.

I have done this in the past when my ONLINEJOBS were gone. Not that I
needed them, but I'd like to keep them around. For me this was caused
by a faulty implementation of the XML writer for onlinejobs. That was
when I re-wrote the XML logic a few months ago.

I don't expect any problems if you proceed your surgery. It could be,
that a couple of prices are missing in case you have added some in the
meantime. Any modification on the reports would also have to be made
again. That's all.

> -------
> 
> I'm doing my monthly reconciliation of all my Merrill Lynch investment  
> accounts, and I started getting the "no price for first use date" for  
> one security.  The equity exists, and there are security accounts for  
> it in two different investmant accounts.  In both cases, the initial  
> presence was from an Add Shares transaction (it was a dividend from a  
> related stock) and was all sold over the past few months.  However,  
> there are currently NO prices for this equity in my file.  Searching  
> through explicit backups and my autosave backups, the prices are in all  
> files prior to 17:15 this evening, but missing from all files after  
> that time.
> 
> Further looking shows that in the earliest bad file, within the  
> <PRICES>......</PRICES> section there is clearly a problems:
> 
>      <PRICEPAIR from="E000520" to="USD">
>        <PRICE date="2023-09-01" price="5159/50" source="Yahoo Finance"/>
>        <PRICE date="2023-09-30" price="10287/100" source="Yahoo  
> Finance"/>
>      </PRICEPAIR>
>    </PRICES>
> </KMYMONEY-FILE>
> <PRICEPAIR from="E000525" to="USD">
>     <PRICE date="2022-06-22" price="2367/50" source="User"/>
>    <PRICE date="2022-09-11" price="2597/50" source="Yahoo Finance"/>
> ....
> 
> Where did that </KMYMONEY-FILE> come from?  I have not been editing the  
> file manually (at least not today :-).

It's not only the </KMYMONEY-FILE> but also the </PRICES> entry. And the
<REPORTS> sections is missing completely. Does that file have a REPORTS
section later on or is it missing completely?

> Stranger, the current file at that point is
>      <PRICEPAIR from="E000520" to="USD">
>        <PRICE date="2022-05-24" price="25080049/380000"  
> source="Transaction"/>
>        <PRICE date="2022-06-03" price="3653/50" source="Yahoo Finance"/>
> ... all these PRICEs are correct
>        <PRICE date="2023-09-30" price="10287/100" source="Yahoo  
> Finance"/>
>      </PRICEPAIR>
>      <PRICEPAIR from="E000525" to="USD">
>        <PRICE date="2022-06-22" price="89971/1940" source="User"/>
>      </PRICEPAIR>
>      <PRICEPAIR from="E000526" to="USD">
>        <PRICE date="2022-06-24" price="20697/380" source="User"/>
>      </PRICEPAIR>
> 
> So it seems to have kept one pricepair for most but not all the  
> following equities, and the prices kept are for the first date of the  
> existing pairs, but the price itself seems totally unrelated.
> 
> Of course since everything past that point is gone, I now have no  
> custom reports either.
> 
> Do I just chalk this up to haveing a bad day, or perhaps sunspots, or  
> is there any hope of figuring out what might have ac tually happened.
>    <REPORTS/>
> One odd thing is that I've got several files with </KMYMONEY-FILE> in  
> the middle of a <PRICEPAIRS> section, so KMM persisted in writing bad  
> files.  All the accounts and transactions seem fine, it's just the  
> prices which are messed up (and the reports missing.)
> 
> Any thoughts or suggestions?
> 
> (The reason I don't want to simply revert to the last known good file  
> is that I'd have to redo over an hour of importing and reconciliation  
> work, although I know I may yet need to resort to that.

The reconciliation data is part of the ACCOUNT elements, so it seems
to be in good condition.

Can you reproduce the problem by using the last good file and repeat what
you have done to get to the first faulty one (in case you remember/can find
out what that was)?

Doesn't KMyMoney complain in case you load a faulty file?

-- 

Regards

Thomas Baumgart

-------------------------------------------------------------
A crash turns an expensive computer into a simple stone!
-------------------------------------------------------------
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 868 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.kde.org/pipermail/kmymoney-devel/attachments/20231003/eda2ded8/attachment.sig>