disappearing data or correupt data file

[Jack]

On 10/3/23 03:25, Thomas Baumgart wrote:
> On Dienstag, 3. Oktober 2023 01:37:32 CEST Jack via KMyMoney-devel wrote:
>> TL;DR at some point in the past day or two (during my monthly
>> reconciliation of my Merrill Lynch investment accounts) a stray
>> </KMYMONEY-FILE> ended up in the middle of the <PRICES> section of my
>> xml data file.  I noticed this because I started getting the
>> consistency-check message about no price on or before the first use of
>> a security.  That led to noticing that many prices were missing, and I
>> ended up actually looking in the xml file.
> That looks weird. As usual, the first question is: which version of KMM
> (stable or master)? The logic to write the XML file differs significantly
> between the two.
Master, compiled within the past few days.  Although I do on occasion 
use 5.1, I don't recall doing so in the past few weeks.  It might be 
interesting if there were some  hint in the file about which version 
saved it.
> A few other questions come to my mind:
> - do you switch between those versions?
> - do you have auto saving enabled?
Only autosave on close if file is changed, 10 versions.
>
>> I'm in process (after making several copies of all saved backups from
>> the past two days) of splicing the missing <PRICEPAIRS> sections the
>> the whole <REPORTS> section from the last good file into the current
>> one (I have no budgets or onlinejobs, so those sections are empty.)
>> I'd love to know how this might have happened, as well as any things to
>> look for that this type of data surgery might introduce.
> I have done this in the past when my ONLINEJOBS were gone. Not that I
> needed them, but I'd like to keep them around. For me this was caused
> by a faulty implementation of the XML writer for onlinejobs. That was
> when I re-wrote the XML logic a few months ago.
>
> I don't expect any problems if you proceed your surgery. It could be,
> that a couple of prices are missing in case you have added some in the
> meantime. Any modification on the reports would also have to be made
> again. That's all.
Good.  That matches what I thought.
>> -------
>>
>> I'm doing my monthly reconciliation of all my Merrill Lynch investment
>> accounts, and I started getting the "no price for first use date" for
>> one security.  The equity exists, and there are security accounts for
>> it in two different investmant accounts.  In both cases, the initial
>> presence was from an Add Shares transaction (it was a dividend from a
>> related stock) and was all sold over the past few months.  However,
>> there are currently NO prices for this equity in my file.  Searching
>> through explicit backups and my autosave backups, the prices are in all
>> files prior to 17:15 this evening, but missing from all files after
>> that time.
>>
>> Further looking shows that in the earliest bad file, within the
>> <PRICES>......</PRICES> section there is clearly a problems:
>>
>>       <PRICEPAIR from="E000520" to="USD">
>>         <PRICE date="2023-09-01" price="5159/50" source="Yahoo Finance"/>
>>         <PRICE date="2023-09-30" price="10287/100" source="Yahoo
>> Finance"/>
>>       </PRICEPAIR>
>>     </PRICES>
>> </KMYMONEY-FILE>
>> <PRICEPAIR from="E000525" to="USD">
>>      <PRICE date="2022-06-22" price="2367/50" source="User"/>
>>     <PRICE date="2022-09-11" price="2597/50" source="Yahoo Finance"/>
>> ....
>>
>> Where did that </KMYMONEY-FILE> come from?  I have not been editing the
>> file manually (at least not today :-).
> It's not only the </KMYMONEY-FILE> but also the </PRICES> entry. And the
> <REPORTS> sections is missing completely. Does that file have a REPORTS
> section later on or is it missing completely?

What I quoted was not all the way to the end of the file.  I'm not at 
home right now, but I believe all the files saved after the event 
included <REPORTS /> and <ONLINEJOBS /> which I read as KMM aborting the 
reading on hitting </KMYMONEY-FILE> and writing a complete file 
including empty sections where it had no data.  However, that does not 
account for the those <PRICEPAIR ...> sections with correct Ennnnn and 
date but wrong amount.  I need to review the autosave files more 
carefully to see what is actually after the </KMYMONEY-FILE>

>> Stranger, the current file at that point is
>>       <PRICEPAIR from="E000520" to="USD">
>>         <PRICE date="2022-05-24" price="25080049/380000"
>> source="Transaction"/>
>>         <PRICE date="2022-06-03" price="3653/50" source="Yahoo Finance"/>
>> ... all these PRICEs are correct
>>         <PRICE date="2023-09-30" price="10287/100" source="Yahoo
>> Finance"/>
>>       </PRICEPAIR>
>>       <PRICEPAIR from="E000525" to="USD">
>>         <PRICE date="2022-06-22" price="89971/1940" source="User"/>
>>       </PRICEPAIR>
>>       <PRICEPAIR from="E000526" to="USD">
>>         <PRICE date="2022-06-24" price="20697/380" source="User"/>
>>       </PRICEPAIR>
>>
>> So it seems to have kept one pricepair for most but not all the
>> following equities, and the prices kept are for the first date of the
>> existing pairs, but the price itself seems totally unrelated.
>>
>> Of course since everything past that point is gone, I now have no
>> custom reports either.
>>
>> Do I just chalk this up to haveing a bad day, or perhaps sunspots, or
>> is there any hope of figuring out what might have ac tually happened.
>>     <REPORTS/>
>> One odd thing is that I've got several files with </KMYMONEY-FILE> in
>> the middle of a <PRICEPAIRS> section, so KMM persisted in writing bad
>> files.  All the accounts and transactions seem fine, it's just the
>> prices which are messed up (and the reports missing.)
>>
>> Any thoughts or suggestions?
>>
>> (The reason I don't want to simply revert to the last known good file
>> is that I'd have to redo over an hour of importing and reconciliation
>> work, although I know I may yet need to resort to that.
> The reconciliation data is part of the ACCOUNT elements, so it seems
> to be in good condition.
It's not so much the actual reconciliation record but my having cleared 
all the transaction, after editing some of them, and actually having had 
to create a few do to having had to remove them manually from the qfx 
file because of wrong security name.
> Can you reproduce the problem by using the last good file and repeat what
> you have done to get to the first faulty one (in case you remember/can find
> out what that was)?
I wish I knew what I had done.  I suppose I can try using the last good 
file and jsut redoing all the reconciliation work since that point.
> Doesn't KMyMoney complain in case you load a faulty file?
I don't recall any complaints, but then when I'm doing that monthly 
reconciliation, I might do over a dozen saves during the process - not 
only one after finishing each account (separate for investment and 
brokerage) but often after clearing all of each different activity 
type.  I'll try again opening a bad file when I get home.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.kde.org/pipermail/kmymoney-devel/attachments/20231003/34938cbf/attachment.htm>