r++ Lexer::token_stream[0]==null_token?

[Roberto Raggi]

Hi Steven!

thanks for reporting it. It is for sure a bug :-) fixed with a slighly 
modified patch #457726

ciao robe

On Tuesday 06 September 2005 14:37, Steven T. Hatton wrote:
> In an effort to solve my problem of determining whether the member variable
> of an AST derivative references a valid token, I modified the Lexer so that
> it creates a "null_token" as the first element of the token_stream.  I also
> modified the Parser::parse() function to advance by
> Parser::token_stream.nextToken() before parsing.  This way, if I use the
> AST node member variable with its default value of 0 as an index into the
> token_stream, I get a "safe" memory location holding an indicator that
> tells me it isn't a valid token form the source code.
>
> void Lexer::tokenize(const char *contents, std::size_t size) {
>   if (!s_initialized)
>     initialize_scan_table();
>
>   token_stream.resize(1024);
>   index = 0;
>
> // My modification -----------
>   Token *current_token = &token_stream[index++];
>   current_token->position = 0;
>   current_token->size = 0;
>   current_token->kind = Token_EOF; // or whatever.
> //--------------------------
>
>   cursor = (const unsigned char *) contents;
>   begin_buffer = (const unsigned char *) contents;
>   end_buffer = cursor + size;
> //...
> }
>
> TranslationUnitAST *Parser::parse(const char *contents, std::size_t size,
> pool *p) {
>   m_no_errors = false;
>   m_pool = p;
>   lexer.tokenize(contents, size);
>   token_stream.nextToken();//My modification
>   TranslationUnitAST *ast = 0;
>   parseTranslationUnit(ast);
>   return ast;
> }
>
> It works in the few tests I've run on it.  But since I really don't know
> how the whole thing is supposed to fit to gether, I don't know if it breaks
> anything.