r++ Lexer::token_stream[0]==null_token?

[Steven T. Hatton]

In an effort to solve my problem of determining whether the member variable of 
an AST derivative references a valid token, I modified the Lexer so that it 
creates a "null_token" as the first element of the token_stream.  I also 
modified the Parser::parse() function to advance by 
Parser::token_stream.nextToken() before parsing.  This way, if I use the AST 
node member variable with its default value of 0 as an index into the 
token_stream, I get a "safe" memory location holding an indicator that tells 
me it isn't a valid token form the source code.

void Lexer::tokenize(const char *contents, std::size_t size) {
  if (!s_initialized)
    initialize_scan_table();

  token_stream.resize(1024);
  index = 0;

// My modification -----------
  Token *current_token = &token_stream[index++];
  current_token->position = 0;
  current_token->size = 0;
  current_token->kind = Token_EOF; // or whatever.
//--------------------------

  cursor = (const unsigned char *) contents;
  begin_buffer = (const unsigned char *) contents;
  end_buffer = cursor + size;
//...
}

TranslationUnitAST *Parser::parse(const char *contents, std::size_t size, pool 
*p) {
  m_no_errors = false;
  m_pool = p;
  lexer.tokenize(contents, size);
  token_stream.nextToken();//My modification
  TranslationUnitAST *ast = 0;
  parseTranslationUnit(ast);
  return ast;
}

It works in the few tests I've run on it.  But since I really don't know how 
the whole thing is supposed to fit to gether, I don't know if it breaks 
anything.
-- 
Regards,
Steven