disappearing data or correupt data file

[Thomas Baumgart]

On Dienstag, 3. Oktober 2023 21:33:07 CEST Jack via KMyMoney-devel wrote:

> On 2023.10.03 15:06, Jack via KMyMoney-devel wrote:
> > First, it seems that the most recent good file is my explicit backup  
> > from 30 Sep and even the oldest autobackup from 02 Oct is bad.  The  
> > main action between them was OFX direct connect download of all  
> > mapped accounts.  Now I'm wondering if there was something funny in  
> > the import.  Actually I know there was something funny, related to  
> > telling KMM not to create the unknown equity, and apparently having  
> > it stop the import (at least for that account) at that point.   
> > However, I don't see how that could create this particular problem.
> > 
> > Comparing XML files is difficult, because (as has been discussed on  
> > the mailing list in the past,) the order of attributes is not  
> > consistent.
> > 
> > xmllint shows as bad only the oldest four of the autobackup files (02  
> > Oct from 16:57 to 17:13.)  I tried using xmllint -c11n, to possibly  
> > be able to diff the various versions, but it won't process the four  
> > "bad" files.
> > 
> > These four files seems to have </KMYMONEY-FILE> inserted immediately  
> > before <PRICEPAIR from="E000525" to="USD"> but otherwise show changes  
> > to the ACCOUNTS and TRANSACTIONS related to my edits as part of  
> > reconciliation.  These all seem to have all the pricepairs intact.   
> > The ends of those files show all my reports, followed by <BUDGETS/>  
> > and <ONLINEJOBS/> but no </KMYMONEY-FILE>.
> I just noticed a second problem, which I should have seen earlier.  Not  
> only has </KMYMONEY-FILE> moved from the end of the file about 400  
> lines up, but </PRICES> has also moved up, from immediately before  
> <REPORTS> to one line above </KMYMONEY-FILE>.
> > 
> > The remaining files have </KMYMONEY-FILE> back to the end of the file  
> > but have the pricepairs after E000525 with only one (wrong) price,  
> > then empty reports, budgets, and online jobs, with the final  
> > </KMYMONEY-FILE>
> > 
> > So, at some point after a big OFX import (and possibly other  
> > (relatively minor?) changes, the last line of the file got moved just  
> > over 400 lines up.  It's likely that the next four files were  
> > sequential saves, without re-reading the file.  I'm guessing that  
> > after saving those four (and probably some earlier ones that got  
> > rolled over with the limit of 10 autobackups) I restarted KMM, and  
> > although it read the bad file, without apparent complaint and saved  
> > valid files after that, it lost all custom reports, and dropped some  
> > pricepars totally and all except one price for the others (after the  
> > pricepair prior to the stray </KMYMONEY-FILE>.
> > 
> > I've just captured console output from a clean start of KMM, reading  
> > first the most recent good file, and then the earliest bad file.   
> > Those logs are identical through loading all the plugins then differe  
> > in the file name (URL) being opened.  There are differences in the  
> > number of items loaded for "P", accounts, "T", "E", all increasing.   
> > There is a decrease in number of prices, even though there are 151  
> > more "<PRICE " entries in the file.    Increase in number of splits.   
> > Differences in time to process home view sections.
> > 
> > The message ' 0 instead of 1 arguments to message "The size of the  
> > file..." supplied before conversion' shows up for both files.
> > 
> > My first attempt at a cut/paste fix didn't work, so I'll try once  
> > more again, otherwise I'll just resign myself to starting over from  
> > the file from the 30th.
> I'll guess now that it didn't work because I didn't correct the  
> misplaced </PRICES> line.

I re-visited this issue and noticed a possibility that could cause the problem.
https://invent.kde.org/office/kmymoney/-/commit/edb22bd2be7dd7c29bd94e14f6a4632866c44ccc
should fix it.

The reason is when erroneous entries are detected during writing (see the
code following the above change which contains continue statements that
cause to reach the m_writer->writeEndElement() line again. Loop twice and you
get </PRICES> in the first iteration and </KMYMONEY-FILE> during the second
one. Only in that case another <PRICEPAIR> can be issued. Resetting the flag
immediately prevents issuing the </PRICES> and </KMYMONEY-FILE> markers
in this situation.

@Jack: I expect those weird price entries to happen during OFX import. So maybe
you can check in that area (OFX prices imported into the last good data file).

-- 

Regards

Thomas Baumgart

-------------------------------------------------------------
As soon as there is an optimist who discovers a light that does not exist
there is a pessimist who blows it out. (Giovanni Guareschi)
-------------------------------------------------------------
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 868 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.kde.org/pipermail/kmymoney-devel/attachments/20231008/bf3c7c7e/attachment.sig>