[Kst] [Bug 120827] Command line option -f fails for some asciifiles

[Ted Kisner]

See below for my proposed patch...

On Monday 30 January 2006 01:09, Brisset, Nicolas wrote:
| I understand the concern that the command line should work, but as I
| wrote a few minutes ago in the bugzilla report, I pretty much like the
| current behavior and I wouldn't want it to be changed too much, as it is
| also critical for kst to be able to accomodate various file formats
| easily (which is the reason I asked for this feature in the very
| beginning quite a while ago, and the need is still here !). 

None of my changes should affect your useage.  The only non-commandline impact 
is that the initial fieldlist in the create new vector dialog is generated 
with the scanning technique.  Apparently you only use text files with a fixed 
number of columns, so you should have no problems.

| What's more, 
| I think it currently works if you set the default settings to the right
| values, and not file-specific values...

Ok, but even if there is a global default, what if I go view a different 
format ascii file and then come back?  I would have to open kst and change 
some global setting every time.  I'm sorry, but having to use the GUI for 
configuration really defeats the useage of kst as a fast tool to view data 
from the commandline- which is what we use it for 90% of the time.

| > Perhaps lines 1, 5, 10, and 50, rather than the 1st 50... but
| > otherwise, I agree.  And it is a significant regression.
| That could be nice for some "auto-detect" mode, but I doubt we can
| always rely on this. Could be the default kst ships with, though...

I think this will be fine for nearly all cases.  Certainly it is better than 
the current case where kst just refuses to display the data.

Here is what this patch does:

1.  In AsciiSource::fieldListFor, if skip (=_dataLines) is non-zero, then the 
code behaves as it does currently.

2.  If skip == 0, then we scan a sampling of (non-comment) lines at the 
beginning of the file.  Currently I scan lines 0, 1, 3, 7, 15, 31, 63, 127 
and take the maximum number of columns found to be the number for the whole 
file.

So if you start kst and go to the data manager and create a new vector from an 
ascii file, then initial fieldlist is populated by this "scanning" procedure.  
If you go to the "configure..." screen and specify a non-zero starting frame, 
then the number of columns is determined from this line alone (like current 
behaviour).

Is this ok to commit?  I also added a FIXME for the future when we might 
consider changing the datasource API.

-Ted


--- kst/datasources/ascii/ascii.cpp     (revision 503725)
+++ kst/datasources/ascii/ascii.cpp     (working copy)
@@ -32,7 +32,7 @@
 #include <qstylesheet.h>

 #include <kcombobox.h>
-#include <kdebug.h>
+#include <ksdebug.h>

 #include <kstmath.h>
 #include "ascii.h"
@@ -605,7 +605,6 @@
   return _numFrames < 1;
 }

-
 QStringList AsciiSource::fieldListFor(const QString& filename, 
AsciiSource::Config *cfg) {
   QStringList rc;
   QFile file(filename);
@@ -651,16 +650,59 @@
   bool done = false;
   QString line;
   int skip = cfg->_dataLine;
-  while (!file.atEnd() && !done) {
+  //FIXME This is a hack which should eventually be fixed by specifying
+  // the starting frame of the data when calling 
KstDataSource::fieldListForSource
+  // and KstDataSource::fieldList.  If the skip value is not specified, then
+  // we scan a few lines and take the maximum number of fields that we find.
+  int maxcnt;
+  if (skip > 0) {
+    maxcnt = -1;
+  } else {
+    maxcnt = 0;
+  }
+  int cnt;
+  int nextscan = 0;
+  int curscan = 0;
+  while (!file.atEnd() && !done && (nextscan < 200)) {
     int r = readFullLine(file, line);
-    if (skip > 0) {
+    if (skip > 0) { //keep skipping until desired line
       --skip;
       if (r < 0) {
         return rc;
       }
       continue;
     }
-    if (r > 1 && !re.exactMatch(line)) {
+    if (maxcnt >= 0) { //original skip value == 0, so scan some lines
+      if (curscan >= nextscan) {
+        if (r > 1 && !re.exactMatch(line)) {
+          line = line.stripWhiteSpace();
+          if (cfg->_columnType == AsciiSource::Config::Custom 
&& !cfg->_columnDelimiter.isEmpty()) {
+            cnt = 
QStringList::split(QRegExp(QString("[%1]").arg(QRegExp::escape(cfg->_columnDelimiter))), 
line, false).count();
+          } else if (cfg->_columnType == AsciiSource::Config::Fixed) {
+            cnt = line.length() / cfg->_columnWidth;
+          } else {
+            cnt = QStringList::split(QRegExp("\\s"), line, false).count();
+          }
+          if (cnt > maxcnt) {
+            maxcnt = cnt;
+          }
+        } else if (r < 0) {
+          return rc;
+        }
+        nextscan += nextscan + 1;
+      }
+      curscan++;
+      continue;
+    }
+    if (r > 1 && !re.exactMatch(line)) { //at desired line, find count
+      line = line.stripWhiteSpace();
+      if (cfg->_columnType == AsciiSource::Config::Custom 
&& !cfg->_columnDelimiter.isEmpty()) {
+        maxcnt = 
QStringList::split(QRegExp(QString("[%1]").arg(QRegExp::escape(cfg->_columnDelimiter))), 
line, false).count();
+      } else if (cfg->_columnType == AsciiSource::Config::Fixed) {
+        maxcnt = line.length() / cfg->_columnWidth;
+      } else {
+        maxcnt = QStringList::split(QRegExp("\\s"), line, false).count();
+      }
       done = true;
     } else if (r < 0) {
       return rc;
@@ -668,17 +710,7 @@
   }

   file.close();
-
-  int cnt;
-  line = line.stripWhiteSpace();
-  if (cfg->_columnType == AsciiSource::Config::Custom 
&& !cfg->_columnDelimiter.isEmpty()) {
-    cnt = 
QStringList::split(QRegExp(QString("[%1]").arg(QRegExp::escape(cfg->_columnDelimiter))), 
line, false).count();
-  } else if (cfg->_columnType == AsciiSource::Config::Fixed) {
-    cnt = line.length() / cfg->_columnWidth;
-  } else {
-    cnt = QStringList::split(QRegExp("\\s"), line, false).count();
-  }
-  for (int i = 1; i <= cnt; ++i) {
+  for (int i = 1; i <= maxcnt; ++i) {
     rc += QString::number(i);
   }