Grammar improvements

[rewinfrey]

Working through ~700 errors in parsing the tensorflow repo. I'll be adjusting the grammar as I work through the errrors, and open this as a WIP PR for visibility and to welcome any suggestions / advice along the way 🙇

/cc @maxbrunsfeld

[rewinfrey]

I based this on https://docs.python.org/3/reference/grammar.html (Python 2's grammar specifies global statements in the same way). To me this reads as a comma separated list of identifiers. Does that seem accurate to you @maxbrunsfeld?

[maxbrunsfeld]

This looks great so far, and your approach (parsing tensorflow) sounds 💯 . Thanks @rewinfrey!

[rewinfrey]

@maxbrunsfeld when we get a chance to 🍐 , I'd like to look at this with you. The nested calls bother me, but in making adjustments to the grammar I haven't found a way to reduce this to a single call.

[maxbrunsfeld]

I agree that there shouldn't be a nested call; this statement should parse as something like

(expression_statement
  (conditional_expression
    (call (identifier) (number) (number) (number))
    (identifier)
    (identifier)))

I think that one problem is that the conditional_expression rule needs an additional expression at the beginning, before the if. From what I understand, conditional expressions have the form

expression1 if condition else expression2

which is similar to the statement form

if condition:
  expression1
else:
  expression2

Currently, you have conditional_expression starting with the keyword if, but that leaves out the expression that should be evaluated if the condition is true.

[rewinfrey]

@maxbrunsfeld thanks for the suggestion! I was trying to model conditional_expression as part of a call expression, but was confused why I was seeing double call in the AST. I've pulled conditional_expression out of call entirely in 1b46c61, so it is modeled in the way you describe. Now the test is structured in the way we'd expect 😄

[maxbrunsfeld]

I don't think these are valid python. Testing them in the Python 2.7 REPL like so:

>>> a = 1
>>> b = 2
>>> if a

I get this:

  File "<stdin>", line 1
    if a
       ^
SyntaxError: invalid syntax

Same with the if/else version. I think some valid examples would be:

doSomething() if someCondition()
doSomething() if someCondition() else doSomethingElse()

In terms of the grammar, I think the first expression in the conditional_expression rule should not be optional.

[rewinfrey]

I think this is ready for review!

Huge 🎩 @maxbrunsfeld for all the advice and 🍐

This parses python2 and python3 grammar rules, and django, hug and tensorflow.

[maxbrunsfeld]

Awesome work, @rewinfrey!