Besides NEWLINE, INDENT and DEDENT, the following categories of tokens
exist: identifiers, keywords, literals,
operators, and delimiters.
Whitespace characters (other than line terminators, discussed earlier)
are not tokens, but serve to delimit tokens.
Where
ambiguity exists, a token comprises the longest possible string that
forms a legal token, when read from left to right.
