Patch for kdevelop-pg to persist the AST

[Jakob Petsovits]

On Monday, 7. August 2006 05:24, Adam Treat wrote:
> Hi Jakob,
>
> Here is a patch that adds a new visitor to the parser which allows
> persistent AST's.  It is called 'serializer'.  The visitor has two public
> static functions: read and write.  Everything else is made private.

Sweet! Compact, readable, applied.

I only made small changes so that token type variables are also serialized, 
and the ast_node's "kind" member as well.
And for the cosmetic changes, I stick with underlines instead of camel casing 
until there's a global option for that.

I'll probably split up the header file into different files soon, I think it's 
better to have AST nodes, parser and visitors seperated.

> Currently, testing with the csharp parser, it serializes an AST where the
> source file was 44K to a 355K file.  This is an uncompressed binary file.
> When I compress it with gzip it goes down to 55K.
>
> I think this exposes a bug in the C# grammar file too.
>
> Line # 1089 of the grammar file has:
>
> member variable declaration_type:
>
> This variable differs with all the other variables and the generated
> serializer needs to comment out this one.

That was a bug in both the grammar and kdev-pg.
kdev-pg was not declaring the variable, and the grammar should have removed 
that variable in the first place. Both are fixed now.

> Jakob, I haven't been able to find a dumptree or something similar in
> kdevelop-pg that would allow me to test the serializer to see if I can get
> the same output from an AST read from the disk as I would from a newly
> created AST via the parser.  How do you test?  Is there a debug mode I
> should know about?

No, there's not AST printer at the moment.
I test the grammar by feeding it with a lot of source files, and when there's 
an error then I know something's wrong. On errors, the current AST nesting is 
printed, so I know approximately where to look.

The Java parser also had AST testing by means of my "javawalk" source file 
reconstructor which is not in SVN but has only been done for the thesis.
(If I can reconstruct the correct Java file, I know that the AST is ok.)

> One more thing... this is all STL and standard lib.  I haven't used Qt.
> Another thing, it does not persist the token stream.  I guess we'll have to
> do this too.

Yup.

> How goes the generated codemodel that uses the AST as a datastore?  Does it
> require a binding pass?

Still missing out on a generated .cpp file, I'll finish and import it later 
today. It doesn't use the AST as a datastore (not at this time), so I'll have 
to do a binder like the C++ one. For now, that is, so I can consider the SoC 
project done. Improvements to that will come after I've got the 
traditionally-generated codemodel.

Thanks for the patch,
  Jakob