Way to analyze docstring and comments about pylint HOT 3 CLOSED

Original comment by Sylvain Thénault (BitBucket: sthenault, GitHub: @sthenault?):

did you take a look at the "raw checker" concept, which are designed for such task where you need low level information not present at the AST level (eg the "format" checker) ?

from pylint.

Original comment by BitBucket: mitar, GitHub: @mitar?:

I did that, but it seems a lot of work. So probably I am not the only one to try to analyze comments as well and I think pylint should have some help for that. And I had some issues getting the whole code to process almost manually.

Here is my current code (not using raw):

https://github.com/wlanslovenija/nodewatcher/blob/development/nodelint.py#L104

So what I would propose is that along with current storing of docstrings, a docstring_raw or something would be stored, which would have """ and all empty lines before and after the docstring.

And same for comments. They could be added to the parse tree as comment (comment content) or comment_raw for including all space before and after the comment, including #.

from pylint.

Closing this one. This can be done currently using the TokenChecker concept. It's a bit of work, yes, but it works. While do want to include the comments at some point into the AST, by bringing a new parser, they will be definitely cleaned up, since the plan is to use them for pragma disables, type comments or just general introspection of the code through the AST. Thus they won't contain the empty lines or spaces before them, this is far more suited to be in the current TokenChecker than in the AST (and they are there right now, by implementing the process_tokens).

from pylint.