`pycep.parser` Module¶

pycep.parser.suite(source, totuple=False)¶

The parser takes a string containing the source code as an input and returns a parse tree.

>>> import pycep.parser
>>> st = pycep.parser.suite('print "Hello, world!"')
>>> st.totuple()
(257, (267, (268, (269, (272, (1, 'print'), (304, (305, (306, (307, (308, (310, (311, (312, (313, (314, (315,     (316, (317, (318, (3, '"Hello, world!"'))))))))))))))))), (4, ''))), (4, ''), (0, ''))

Formally, this is an LL(2) (Left-to-right, Leftmost derivation with two-token lookahead), recursive-descent parser.

Parameters:	source (string) – Source code totuple (boolean) – (for testing) Return internal parser data structure, don’t convert to `parser.st` object
Returns:	Parse Tree
Return type:	parser.st
Raises:	`SyntaxError` – Syntax error in the source code

Parse Tree of Hello World Example:

See also

Python Language Reference: https://docs.python.org/2/reference/grammar.html
Non-Terminal Symbols: https://hg.python.org/cpython/file/2.7/Lib/symbol.py
Leaf Nodes: https://docs.python.org/2/library/token.html
LL Parser: https://en.wikipedia.org/wiki/LL_parser
Recursive-Descent Parser: https://en.wikipedia.org/wiki/Recursive_descent_parser
Future Statement Definitions: https://docs.python.org/2/library/__future__.html

pycep.parser.listit(tup)¶: Recursively convert list-of-lists to tuples-of-tuples

pycep.parser._single_input(tokens)¶

Parse a single input.

single_input: NEWLINE | simple_stmt | compound_stmt NEWLINE

pycep.parser._file_input(tokens)¶

Parse a module or sequence of command read from an input file.

file_input: (NEWLINE | stmt)* ENDMARKER

pycep.parser._eval_input(tokens)¶

Parse an evaluation input.

eval_input: testlist NEWLINE* ENDMARKER

pycep.parser._decorator(tokens)¶

Parse a decorator.

decorator: '@' dotted_name [ '(' [arglist] ')' ] NEWLINE

pycep.parser._decorators(tokens)¶

Parse a list of decorators.

decorators: decorator+

pycep.parser._decorated(tokens)¶

Parse a decorated statement.

decorated: decorators (classdef | funcdef)

pycep.parser._funcdef(tokens)¶

Parse a function definition.

funcdef: 'def' NAME parameters ':' suite

pycep.parser._parameters(tokens)¶

Parse a parameter list.

parameters: '(' [varargslist] ')'

pycep.parser._varargslist(tokens)¶

Parse a variable argument list.

varargslist: ((fpdef ['=' test] ',')*
              ('*' NAME [',' '**' NAME] | '**' NAME) |
              fpdef ['=' test] (',' fpdef ['=' test])* [','])

pycep.parser._fpdef(tokens)¶

Parse function parameter definition.

fpdef: NAME | '(' fplist ')'

pycep.parser._fplist(tokens)¶

Parse a function parameter list

fplist: fpdef (',' fpdef)* [',']

pycep.parser._stmt(tokens)¶

Parse a statement.

stmt: simple_stmt | compound_stmt

pycep.parser._simple_stmt(tokens)¶

Parse a simple statement.

simple_stmt: small_stmt (';' small_stmt)* [';'] NEWLINE

pycep.parser._small_stmt(tokens)¶

Parse a small statement.

small_stmt: (expr_stmt | print_stmt | del_stmt | pass_stmt | flow_stmt |
             import_stmt | global_stmt | exec_stmt | assert_stmt)

pycep.parser._expr_stmt(tokens)¶

Parse an expr stmt.

expr_stmt: testlist (augassign (yield_expr|testlist) |
                     ('=' (yield_expr|testlist))*)

pycep.parser._augassign(tokens)¶

Parse an augmented assign statement.

augassign: ('+=' | '-=' | '*=' | '/=' | '%=' | '&=' | '|=' | '^=' |
'<<=' | '>>=' | '**=' | '//=')

pycep.parser._print_stmt(tokens)¶

Parse a print statement.

print_stmt: 'print' ( [ test (',' test)* [','] ] |
                      '>>' test [ (',' test)+ [','] ] )

pycep.parser._del_stmt(tokens)¶

Parse a delete statement.

del_stmt: 'del' exprlist

pycep.parser._pass_stmt(tokens)¶

Parse a pass statement.

pass_stmt: 'pass'

pycep.parser._flow_stmt(tokens)¶

Parse a flow statement.

flow_stmt: break_stmt | continue_stmt | return_stmt | raise_stmt | yield_stmt

pycep.parser._break_stmt(tokens)¶

Parse a break statement.

break_stmt: 'break'

pycep.parser._continue_stmt(tokens)¶

Parse a continue statement.

continue_stmt: 'continue'

pycep.parser._return_stmt(tokens)¶

Parse a return statement.

return_stmt: 'return' [testlist]

pycep.parser._yield_stmt(tokens)¶

Parse a yield statement.

yield_stmt: yield_expr

pycep.parser._raise_stmt(tokens)¶

Parse a raise statement.

raise_stmt: 'raise' [test [',' test [',' test]]]

pycep.parser._import_stmt(tokens)¶

Parse an import statement.

import_stmt: import_name | import_from

pycep.parser._import_name(tokens)¶

Parse an import name.

import_name: 'import' dotted_as_names

pycep.parser._import_from(tokens)¶

Parse an import from.

import_from: ('from' ('.'* dotted_name | '.'+)
              'import' ('*' | '(' import_as_names ')' | import_as_names))

pycep.parser._import_as_name(tokens)¶

Parse an import as names.

import_as_name: NAME ['as' NAME]

pycep.parser._dotted_as_name(tokens)¶

Parse a dotted as name.

dotted_as_name: dotted_name ['as' NAME]

pycep.parser._import_as_names(tokens)¶

Parse import as names.

import_as_names: import_as_name (',' import_as_name)* [',']

pycep.parser._dotted_as_names(tokens)¶

Parse dotted as names.

dotted_as_names: dotted_as_name (',' dotted_as_name)*

pycep.parser._dotted_name(tokens)¶

Parse a dotted name.

dotted_name: NAME ('.' NAME)*

pycep.parser._global_stmt(tokens)¶

Parse a global statement.

global_stmt: 'global' NAME (',' NAME)*

pycep.parser._exec_stmt(tokens)¶

Parse an exec statement.

exec_stmt: 'exec' expr ['in' test [',' test]]

pycep.parser._assert_stmt(tokens)¶

Parse an assert statement.

assert_stmt: 'assert' test [',' test]

pycep.parser._compound_stmt(tokens)¶

Parse a compound statement.

compound_stmt: if_stmt | while_stmt | for_stmt | try_stmt | with_stmt | funcdef | classdef | decorated

pycep.parser._if_stmt(tokens)¶

Parse and if statement.

if_stmt: 'if' test ':' suite ('elif' test ':' suite)* ['else' ':' suite]

pycep.parser._while_stmt(tokens)¶

Parse a while statement.

while_stmt: 'while' test ':' suite ['else' ':' suite]

pycep.parser._for_stmt(tokens)¶

Parse a for statement.

for_stmt: 'for' exprlist 'in' testlist ':' suite ['else' ':' suite]

pycep.parser._try_stmt(tokens)¶

Parse a try statement.

try_stmt: ('try' ':' suite
           ((except_clause ':' suite)+
            ['else' ':' suite]
            ['finally' ':' suite] |
           'finally' ':' suite))

pycep.parser._with_stmt(tokens)¶

Parse a with statement.

with_stmt: 'with' with_item (',' with_item)*  ':' suite

pycep.parser._with_item(tokens)¶

Parse a with item.

with_item: test ['as' expr]

pycep.parser._except_clause(tokens)¶

Parse an except clause.

except_clause: 'except' [test [('as' | ',') test]]

pycep.parser._suite(tokens)¶

Parse a suite.

suite: simple_stmt | NEWLINE INDENT stmt+ DEDENT

pycep.parser._testlist_safe(tokens)¶

Parse a testlist safe.

testlist_safe: old_test [(',' old_test)+ [',']]

pycep.parser._old_test(tokens)¶

Parse an old test.

old_test: or_test | old_lambdef

pycep.parser._old_lambdef(tokens)¶

Parse an old lambda definition.

old_lambdef: 'lambda' [varargslist] ':' old_test

pycep.parser._test(tokens)¶

Parse a test statement.

test: or_test ['if' or_test 'else' test] | lambdef

pycep.parser._or_test(tokens)¶

Parse an or_test statement

or_test: and_test ('or' and_test)*

pycep.parser._and_test(tokens)¶

Parse an and test statement.

and_test: not_test ('and' not_test)*

pycep.parser._not_test(tokens)¶

Parse a not test statement.

not_test: 'not' not_test | comparison

pycep.parser._comparison(tokens)¶

Parse a comparison.

comparison: expr (comp_op expr)*

pycep.parser._comp_op(tokens)¶

Parse a compare operator statement.

comp_op: '<'|'>'|'=='|'>='|'<='|'<>'|'!='|'in'|'not' 'in'|'is'|'is' 'not'

pycep.parser._expr(tokens)¶

Parse an expression statement.

expr: xor_expr ('|' xor_expr)*

pycep.parser._xor_expr(tokens)¶

Parse an xor expression statement.

xor_expr: and_expr ('^' and_expr)*

pycep.parser._and_expr(tokens)¶

Parse an and expression statement.

and_expr: shift_expr ('&' shift_expr)*

pycep.parser._shift_expr(tokens)¶

Parse a shift_expr statement

shift_expr: arith_expr (('<<'|'>>') arith_expr)*

pycep.parser._arith_expr(tokens)¶

Parse an arithmetic expression statement.

arith_expr: term (('+'|'-') term)*

pycep.parser._term(tokens)¶

Parse a term statement.

term: factor (('*'|'/'|'%'|'//') factor)*

pycep.parser._factor(tokens)¶

Parse a factor statement.

factor: ('+'|'-'|'~') factor | power

pycep.parser._power(tokens)¶

Parse a power statement.

power: atom trailer* ['**' factor]

pycep.parser._atom(tokens)¶

Parse an atom statement.

atom: ('(' [yield_expr|testlist_comp] ')' |
       '[' [listmaker] ']' |
       '{' [dictorsetmaker] '}' |
       '`' testlist1 '`' |
       NAME | NUMBER | STRING+)

pycep.parser._listmaker(tokens)¶

Parse a listmaker statement.

listmaker: test ( list_for | (',' test)* [','] )

pycep.parser._testlist_comp(tokens)¶

Parse a testlist comp statement.

testlist_comp: test ( comp_for | (',' test)* [','] )

pycep.parser._lambdef(tokens)¶

Parse a lambda definition.

lambdef: 'lambda' [varargslist] ':' test

pycep.parser._trailer(tokens)¶

Parse a trailer.

trailer: '(' [arglist] ')' | '[' subscriptlist ']' | '.' NAME

pycep.parser._subscriptlist(tokens)¶

Parse a subscriptlist.

subscriptlist: subscript (',' subscript)* [',']

pycep.parser._subscript(tokens)¶

Parse a subscript.

subscript: '.' '.' '.' | test | [test] ':' [test] [sliceop]

pycep.parser._sliceop(tokens)¶

Parse a slice operation.

sliceop: ':' [test]

pycep.parser._exprlist(tokens)¶

Parse an expression list.

exprlist: expr (',' expr)* [',']

pycep.parser._testlist(tokens)¶

Parse a testlist.

testlist: test (',' test)* [',']

pycep.parser._dictorsetmaker(tokens)¶

Parse a dict or set maker statement.

dictorsetmaker: ( (test ':' test (comp_for | (',' test ':' test)* [','])) |
                  (test (comp_for | (',' test)* [','])) )

pycep.parser._classdef(tokens)¶

Parse a class definition.

classdef: 'class' NAME ['(' [testlist] ')'] ':' suite

pycep.parser._arglist(tokens)¶

Parse an argument list.

arglist: (argument ',')* (argument [',']
                         |'*' test (',' argument)* [',' '**' test]
                         |'**' test)

pycep.parser._argument(tokens)¶

Parse an argument.

argument: test [comp_for] | test '=' test

pycep.parser._list_iter(tokens)¶

Parse a list iteration.

list_iter: list_for | list_if

pycep.parser._list_for(tokens)¶

Parse a list for.

list_for: 'for' exprlist 'in' testlist_safe [list_iter]

pycep.parser._list_if(tokens)¶

Parse a list if.

list_if: 'if' old_test [list_iter]

pycep.parser._comp_iter(tokens)¶

Parse a generator expression.

comp_iter: comp_for | comp_if

pycep.parser._comp_for(tokens)¶

Parse a for generator expression.

comp_for: 'for' exprlist 'in' or_test [comp_iter]

pycep.parser._comp_if(tokens)¶

Parse an if generator expression.

comp_if: 'if' old_test [comp_iter]

pycep.parser._testlist1(tokens)¶

Parse a testlist1.

testlist1: test (',' test)*

pycep.parser._encoding_decl(tokens)¶

Parse an encoding declaration.

encoding_decl: NAME

pycep.parser._yield_expr(tokens)¶

Parse a yield expression.

yield_expr: 'yield' [testlist]

class pycep.parser.TokenIterator(generator, skip_tokens=(54, 53))¶

Wrapper for token generator which allows checking for and accepting tokens. Physical line breaks (tokenize.NL) and comments (token.N_TOKENS) are skipped from the input.

check_test(lookahead=1)¶: Shorthand notation to check whether next statement is a test

check_comp_op()¶: Shorthand notation to check whether next statement is a comp_op

check_expr(lookahead=1)¶: Shorthand notation to check whether next statement is an expr

comments powered by Disqus

pycep.parser Module¶

`pycep.parser` Module¶