kdev-pg: Token definitions

[Jakob Petsovits]

On Friday, 28. October 2005 21:05, Roberto Raggi wrote (on IRC):
<roberto__> jpetso: the token is declared when it is used the first time
<roberto__> jpetso: i see it's a problem if you already have a lexer
<roberto__> jpetso: i can probably improve it

...or for those who haven't used kdevelop-pg yet:
There are no token declarations in the grammar file. Every time when a token 
is used in a rule, it is automatically declared. This is a problem, because

a) When you want to test the lexer, you have to do a token rule, like
   ( ABSTRACT | ASSERT | BOOLEAN | BREAK | BYTE | CASE | CATCH | ... )
-> tokens ;;
Without this the lexer won't compile because of missing declarations.

b) What's worse than this is that you can easily make errors. Say your lexer 
returns the LESS_THAN token for all '<' characters, but out of lazyness you 
don't remember it right and write a rule
   ( expression LT expression )
-> less_than_expression ;;
This rule will never match, because there is no LT token. Nevertheless, it is 
declared, and kdevelop-pg doesn't even give a warning.


The attached patch tries to solve this by adding a new token declaration 
syntax for grammar files. Each token has to be defined there, or you get an 
error when translating the grammar with kdev-pg. I made this syntax for it:
%token ABSTRACT, ASSERT, BOOLEAN, BREAK, BYTE, CASE, CATCH, ... ;;
You can also have more than one of these. Each one can span multiple lines.
Of course, if we want it to look different, that's no problem, just tell me.

The adapted grammar parser only calls push_terminal() (which adds the tokens 
to the defined list) for these constructs. For normal token usages inside 
rules it calls the new _G_system.terminal() method which retrieves either the 
defined token or a new, undefined one. The undefined ones don't get far 
because they are caught by an undefined_token_checker class which exits with 
an error.

Is the patch clean enough to commit?
Of course, the grammars will be adapted too.

Wishes,
  Jakob
-------------- next part --------------
A non-text attachment was scrubbed...
Name: kdev-pg-token-defs.diff.tar.gz
Type: application/x-tgz
Size: 1761 bytes
Desc: not available
URL: <http://mail.kde.org/pipermail/kdevelop-devel/attachments/20060207/f4f3b718/attachment.bin>