3.2 pyggy.lexer - The lexing engine

pyggy.lexer.TOK_ERR
TOK_ERR is returned by the lexer whenever a character is encountered which cannot be lexed.

class pyggy.lexer.lexer( lexspec)
The lexer class provides an extensible lexer class capable of lexing tokens from an input source based on tables generated by PyLly. The lexspec argument is passed in from a generated table module. It should be a tuple made up of a DFA table, a list of start states, a list of actions for each accepting state, a list of actions to be performed at the end of file and a dictionary of character classes. The format of the lexspec argument is subject to change.

The lexer class provides lexing from input strings or from files. The class can be subclassed to provide lexers with different input behaviors.

value
The value member holds the value associated with the most recently returned token.

setinputstr( str)
Sets the input source to be the characters in the str argument.

setinput( fname)
Sets the input to source to be the characters from the specified file. A filename of - indicates data should come from stdin.

token( )
Returns the next token lexed from the input stream. If there are no more tokens left, the value None is returned. If an error occurs the value pyggy.lexer.TOK_EOF is returned. In the future an exception may be raised in this situation. Before a token is returned, the value member is set to the value specified in the action code for the token, or to a string of the characters in the token.

See the PyGgy Home Page.