Option to "tail -f" ASCII file

[Barth Netterfield]

Are you in "read to end" mode or in "count from end" mode?

CBN

On Sun, Mar 3, 2019, 12:08 PM Scott Armitage, <scott at armitage.space> wrote:

> At a data source file size of ~17.5 MiB, Kst2.exe is consuming ~80% CPU. I
> am now confident that it is _not_ a memory issue, as Kst2.exe is only using
> ~100 MiB of memory.
>
> -S
>
>
> On 2 Mar 2019, at 13:09, Scott Armitage <scott at armitage.space> wrote:
>
> At ~10 MiB, kst2.exe is now pulling a constant 40% to 50% CPU usage.
>
> -S
>
> On 2 Mar 2019, at 11:45, Scott Armitage <scott at armitage.space> wrote:
>
> For example, by the time my current log file had reached ~3.5 MiB, Kst2
> was using nearly 30% of the CPU (Core i5-8250) and UI response of Kst2 had
> become sluggish. Responsiveness of the rest of the system remains fine.
>
> -S
>
>
> On 2 Mar 2019, at 10:14, Scott Armitage <scott at armitage.space> wrote:
>
> Hi Nicolas et al,
>
> I would be surprised if I am hitting memory limits, but I will check the
> next time I observe the slow-down (which should be this weekend, as I have
> just started a 48-hour test). I currently rotate the log files at 50 MiB,
> but that number is arbitrary; however, the lower I make it, the more often
> I have to go in and change the data source file.
>
> The ASCII files (comma-delimited CSV, with either Windows or UNIX line
> endings depending on the exact situation) start empty. There are typically
> 28 columns, the first of which is an ISO timestamp of the form “yyyy-mm-dd
> HH:MM:SS.zzzzzz”, which I parse in Kst as the timestamp for each row. All
> other columns are floating point numerical data, which may occasionally
> read `nan`, `+inf`, or `-inf`, except for 7 of the columns which are
> integer numerical data.
>
> In typical usage, I am parsing 12 of the columns, plus the timestamp, and
> plotting them in four plots. The data is being plotted with lines, not dots.
>
> The log file is being written to at a row cadence of ~0.4 s from Python,
> calling `file.flush()` after each row to allow Kst to see the update.
>
> Is it the “Limit Buffer Size” option you are referring to? I have tried
> playing around with that, setting it to various values like 5 MiB or 10
> MiB. I haven’t seen a benefit from it yet. When Kst starts to feel
> sluggish, its CPU usage goes to 25% (i.e. maxing out one full virtual core
> on my machine).
>
> In case it is relevant, I am running Kst2 32-bit on Windows 10 64-bit.
>
> Cheers,
> -S
>
> p.s. I have subscribed to the mailing list, so I should get responses to
> there now.
>
>
> *De :* BRISSET, Nicolas
> *Envoyé :* vendredi 1 mars 2019 18:13
> *À :* 'kst at kde.org'
> *Objet :* RE: Option to "tail -f" ASCII file
>
> Hi,
>
> Kst does not reload the files entirely, it parses them and keeps a pointer
> to the last processed line, and resumes from there in case new data arrives.
> It may become slow with large amounts of data, your best bet is to tune
> the memory options available in the data wizard to avoid using up too much
> memory, which could lead your PC to swapping and that could make it slow.
> What amount of data (size of ASCII file, number of columns, number of
> lines) are you dealing with, and in which form are they ploted (lines are
> much better optimized than points)?
>
> Nicolas
>
> *De :* Kst [mailto:kst-bounces at kde.org <kst-bounces at kde.org>] *De la part
> de* Scott Armitage
> *Envoyé :* jeudi 28 février 2019 17:27
> *À :* kst at kde.org
> *Objet :* Option to "tail -f" ASCII file
>
> Is there an option to follow ASCII files instead of reloading them in
> their entirety? Would this be a relatively straightforward addition?
>
> I have a set of Python tools that generate CSV log files of telemetry,
> which are used by post-processing analysis tools. I recently started using
> Kst2 to visualize the data in real time. This works excellently! However,
> as the CSV files grow, Kst2 becomes sluggish as it reloads the entire file
> as new lines are written. In my environment, I know that these are output
> files that only grow, like log files. An option to follow the files,
> similar to "tail -f", would I think dramatically improve Kst2's performance
> in plotting these files in real time.
> The information in this e-mail is confidential. The contents may not be
> disclosed or used by anyone other than the addressee. Access to this e-mail
> by anyone else is unauthorised.
> If you are not the intended recipient, please notify Airbus immediately
> and delete this e-mail.
> Airbus cannot accept any responsibility for the accuracy or completeness
> of this e-mail as it has been sent over public networks. If you have any
> concerns over the content of this message or its Accuracy or Integrity,
> please contact Airbus immediately.
> All outgoing e-mails from Airbus are checked using regularly updated virus
> scanning software but you should take whatever measures you deem to be
> appropriate to ensure that this message and any attachments are virus free.
>
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.kde.org/pipermail/kst/attachments/20190303/ca91a1f8/attachment-0001.html>