Third iteration of QMake parser, looking for a parser generator

[Roberto Raggi]

Hi Andreas,

Il giorno 05/lug/07, alle ore 13:55, Andreas Pakulat ha scritto:
> So to conclude: kdev-pg will get another use-case :)

This is a very good news for KDevelop. You can help the kdev  
developers working on kdev-pg to improve it ;-) I really think it  
should be easy to change kdev-pg to generate the code you need for  
the qmake parser and I'm sure that those changes will be very helpful  
for the other programming language plug-ins.

>
> That would be cool, I've already got an idea how to write it, but that

You can find in attachment a little example of an incremental lexer  
for a dummy language. The example is small (about 200 loc), but it  
shows how to recognize common tokens like whitespaces, C++ comments,  
string literals, identifiers, and operators with 1 or 2 characters.  
It also has 2 states, so you have an example of how to handle multi- 
state lexers.

The signature of the Lexer is very simple, so you have two methods to  
get and set the active state, a method that returns the type of the  
current token and a method(it is a function call operator actually)  
that advance the scanner.

class Lexer: Constants {
public:
   Lexer(int state = State_default);

   inline int state() const { return _state; }

   inline void setState(int state) { _state = state; }

   inline int token() const { return _token; }

   const QChar *operator()(const QChar *it, const QChar *last);

  ..

};

how do you use it? well, if you need to scan a file you do

void tokenize(const QString &filename) {
   QFile file(filename);

   if (file.open(QFile::ReadOnly)) {
     QTextStream stream(&file);
     QString contents = stream.readAll();

     const QChar *begin = contents.unicode();
     const QChar *it = begin, *end = begin + contents.length();

     Lexer yylex;

     while (it != end) {
       int start = it - begin;

       it = yylex(it, end);

       int stop = it - begin;

       qDebug() << "*** token:" << yylex.token() << "location:" <<  
start << stop;
     }
   }
}

or if you want to use it as syntax highlighter..

class SyntaxHighlighter: public QSyntaxHighlighter, Constants {
public:
   SyntaxHighlighter(QTextEdit *editor):
     QSyntaxHighlighter(editor) {}

protected:
   virtual void highlightBlock(const QString &text) {

     int state = previousBlockState();

     yylex.setState(state != -1 ? state : State_default);

     const QChar *begin = text.unicode();
     const QChar *it = begin, *end = begin + text.length();

     while (it != end) {
       int start = it - begin;
       it = yylex(it, end);
       int stop = it - begin;

       switch (yylex.token()) {
       case Token_comment:
	setFormat(start, stop - start, Qt::darkRed);
	break;

        /// more ...

       default:
	break;
       }
     }

     setCurrentBlockState(yylex.state());
   }

private:
   Lexer yylex;
};


as you can see it is very simple to use. You can generate the code to  
recognize the keywords with kwgen.cpp the little tool I posted 2-3  
days ago. I think you should be able to change the example to  
recognize tokens for qmake or any other other(real?) language.

ciao robe
-------------- next part --------------
A non-text attachment was scrubbed...
Name: lex.cpp
Type: application/octet-stream
Size: 4130 bytes
Desc: not available
URL: <http://mail.kde.org/pipermail/kdevelop-devel/attachments/20070705/482c1abd/attachment.obj>
-------------- next part --------------