Add comments to the AST about rust_hdl HOT 7 CLOSED

I had naively started to implement (1) but I like your solution (3) much better. I'll let you know once I've got something worth looking at.

from rust_hdl.

Thinking about how to deal with comments at the end of the file. There won't be a token that it makes sense to associate them with. I'll just create a FinalComment token. Since this can only occur in one place, the parsing change will be minor.

from rust_hdl.

I've been thinking about this some more (and trying to take the next step to moving the comments from the tokens to the AST) and I'm not happy about how I have implemented it so far. In the current implementation I assigned all comments to a token. Most positions don't make any sense and we need to gather up all those comments to either move them or emit warnings for them. This gathering gets messy.

I would like to remove the comments from the tokens and implement the following methods on the Tokenizer:

// Gets the leading comments for the next token.
pub fn get_leading_comments(&mut self) -> Vec(Comment);
// Gets the trailing comment for the previous token.
pub fn get_trailing_comment(&mut self) -> Option(Comment);
// Returns a vector of all the comments that were parsed since we last
// handled.  Does not include comments that were
// retrieved using get_leading_comments or get_trailing_comment.  This
// is useful if we wish to move all these comments to a new location or to
// generate warning messages from them.
pub fn handle_new_comments(&mut self) -> Vec(Comment);

So if we were parsing a type_definition:

1. Assert that there aren't any already parsed comments that haven't been handled.
2. Get the leading comments for the definition.
3. Parse the definition.
4. Get the trailing comment.
5. Get the comments that weren't handled. Either combine them into the other comments, or ignore them. Generate warning messages either way.
6. Add the combined comment (probably a latin_1_str) into the type_definition object.

The main complexity is dealing with set_state since that should modify the vector of unhandled comments. We could replace set_state with move_backwards which just drops any unhandled comments from the list if their position is ahead of the new state. The other main use of set_state is in move_after(token) which seems to be exclusively used for moving after a peeked token. This could be replaced with move_after_peek() which just moves after the token previously peeked. The state and unhandled comments could be stored in the Tokenizer whenever a peek is performed.

Does this seem like a reasonable approach?

from rust_hdl.

I agree with the approach.

from rust_hdl.

I've reworked the comment parsing along these lines in #30.
I'm still working on improving the tests and getting the comments into the AST.

from rust_hdl.

@benreynwar what has happened with the PR, what is the current status?

from rust_hdl.

It's abandoned. I'm trying to focus my 'fun work' time on ghdl now.

from rust_hdl.