Option to "tail -f" ASCII file

[Scott Armitage]

Hi Nicolas et al,

I would be surprised if I am hitting memory limits, but I will check the next time I observe the slow-down (which should be this weekend, as I have just started a 48-hour test). I currently rotate the log files at 50 MiB, but that number is arbitrary; however, the lower I make it, the more often I have to go in and change the data source file.

The ASCII files (comma-delimited CSV, with either Windows or UNIX line endings depending on the exact situation) start empty. There are typically 28 columns, the first of which is an ISO timestamp of the form “yyyy-mm-dd HH:MM:SS.zzzzzz”, which I parse in Kst as the timestamp for each row. All other columns are floating point numerical data, which may occasionally read `nan`, `+inf`, or `-inf`, except for 7 of the columns which are integer numerical data.

In typical usage, I am parsing 12 of the columns, plus the timestamp, and plotting them in four plots. The data is being plotted with lines, not dots.

The log file is being written to at a row cadence of ~0.4 s from Python, calling `file.flush()` after each row to allow Kst to see the update.

Is it the “Limit Buffer Size” option you are referring to? I have tried playing around with that, setting it to various values like 5 MiB or 10 MiB. I haven’t seen a benefit from it yet. When Kst starts to feel sluggish, its CPU usage goes to 25% (i.e. maxing out one full virtual core on my machine).

In case it is relevant, I am running Kst2 32-bit on Windows 10 64-bit.

Cheers,
-S

p.s. I have subscribed to the mailing list, so I should get responses to there now.
 
> De : BRISSET, Nicolas 
> Envoyé : vendredi 1 mars 2019 18:13
> À : 'kst at kde.org <mailto:kst at kde.org>'
> Objet : RE: Option to "tail -f" ASCII file
>  
> Hi,
>  
> Kst does not reload the files entirely, it parses them and keeps a pointer to the last processed line, and resumes from there in case new data arrives.
> It may become slow with large amounts of data, your best bet is to tune the memory options available in the data wizard to avoid using up too much memory, which could lead your PC to swapping and that could make it slow.
> What amount of data (size of ASCII file, number of columns, number of lines) are you dealing with, and in which form are they ploted (lines are much better optimized than points)?
>  
> Nicolas
>  
> De : Kst [mailto:kst-bounces at kde.org <mailto:kst-bounces at kde.org>] De la part de Scott Armitage
> Envoyé : jeudi 28 février 2019 17:27
> À : kst at kde.org <mailto:kst at kde.org>
> Objet : Option to "tail -f" ASCII file
>  
> Is there an option to follow ASCII files instead of reloading them in their entirety? Would this be a relatively straightforward addition?
>  
> I have a set of Python tools that generate CSV log files of telemetry, which are used by post-processing analysis tools. I recently started using Kst2 to visualize the data in real time. This works excellently! However, as the CSV files grow, Kst2 becomes sluggish as it reloads the entire file as new lines are written. In my environment, I know that these are output files that only grow, like log files. An option to follow the files, similar to "tail -f", would I think dramatically improve Kst2's performance in plotting these files in real time.
> The information in this e-mail is confidential. The contents may not be disclosed or used by anyone other than the addressee. Access to this e-mail by anyone else is unauthorised.
> If you are not the intended recipient, please notify Airbus immediately and delete this e-mail.
> Airbus cannot accept any responsibility for the accuracy or completeness of this e-mail as it has been sent over public networks. If you have any concerns over the content of this message or its Accuracy or Integrity, please contact Airbus immediately.
> All outgoing e-mails from Airbus are checked using regularly updated virus scanning software but you should take whatever measures you deem to be appropriate to ensure that this message and any attachments are virus free.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.kde.org/pipermail/kst/attachments/20190302/7cc37a91/attachment.html>