; "Ic@sWdZdZdZddlZddlZddlZddlTddlmZddl m Z m Z ej dZ ddlZd eeDd d d d ddgZ[eZd ee6e:d?6e9d@6e:dA6e9dB6e:dC6e9dD6e:dE6e9dF6e:dG6e9dH6e:dI6e9dJ6e:dK6e9dL6e:dM6ddN6ddO6ddP6ddQ6Z<iZ=xdnD]Z>e>e=e>e>e?e>upGvR, ESR, Tim Peters, Thomas Wouters, Fred Drake, Skip Montanaro, Raymond Hettinger, Trent Nelson, Michael FoordiN(u*(ulookup(uchainurepeatucoding[:=]\s*([-\w.]+)cCs0g}|]"}|ddko ||q q S(iu_((u.0u_[1]ux((u%/mit/python/lib/python3.0/tokenize.pyu "s uCOMMENTutokenizeudetect_encodinguNLu untokenizeuENCODINGiiicGsddj|dS(Nu(u|u)(ujoin(uchoices((u%/mit/python/lib/python3.0/tokenize.pyugroup.scGst|dS(Nu*(ugroup(uchoices((u%/mit/python/lib/python3.0/tokenize.pyuany/scGst|dS(Nu?(ugroup(uchoices((u%/mit/python/lib/python3.0/tokenize.pyumaybe0su[ \f\t]*u #[^\r\n]*u\\\r?\nu [a-zA-Z_]\w*u0[xX][0-9a-fA-F]+u 0[bB][01]+u 0[oO][0-7]+u(?:0+|[1-9][0-9]*)u[eE][-+]?[0-9]+u[0-9]+\.[0-9]*u\.[0-9]+u[0-9]+u [0-9]+[jJ]u[jJ]u[^'\\]*(?:\\.[^'\\]*)*'u[^"\\]*(?:\\.[^"\\]*)*"u%[^'\\]*(?:(?:\\.|'(?!''))[^'\\]*)*'''u%[^"\\]*(?:(?:\\.|"(?!""))[^"\\]*)*"""u [bB]?[rR]?'''u [bB]?[rR]?"""u&[bB]?[rR]?'[^\n'\\]*(?:\\.[^\n'\\]*)*'u&[bB]?[rR]?"[^\n"\\]*(?:\\.[^\n"\\]*)*"u\*\*=?u>>=?u<<=?u!=u//=?u->u[+\-*/%&|^=<>]=?u~u[][(){}]u\r?\nu\.\.\.u[:;.,@]u%[bB]?[rR]?'[^\n'\\]*(?:\\.[^\n'\\]*)*u'u%[bB]?[rR]?"[^\n"\\]*(?:\\.[^\n"\\]*)*u"u'''u"""ur'''ur"""ub'''ub"""ubr'''ubr"""uR'''uR"""uB'''uB"""ubR'''ubR"""uBr'''uBr"""uBR'''uBR"""uruRubuBur'ur"uR'uR"ub'ub"uB'uB"ubr'ubr"uBr'uBr"ubR'ubR"uBR'uBR"icBs|EeZdS(N(u__name__u __module__(u __locals__((u%/mit/python/lib/python3.0/tokenize.pyu TokenErrors u TokenErrorcBs|EeZdS(N(u__name__u __module__(u __locals__((u%/mit/python/lib/python3.0/tokenize.pyuStopTokenizings uStopTokenizingcBs2|EeZdZdZdZdZdS(cCs(g|_d|_d|_d|_dS(Nii(utokensuprev_rowuprev_coluNoneuencoding(uself((u%/mit/python/lib/python3.0/tokenize.pyu__init__s   cCs<|\}}||j}|o|jjd|ndS(Nu (uprev_colutokensuappend(uselfustarturowucolu col_offset((u%/mit/python/lib/python3.0/tokenize.pyuadd_whitespaces  cCsx|D]}t|dko|j||Pn|\}}}}}|tko||_qn|j||jj||\|_|_|t t fko|jd7_d|_qqWdj |jS(Niiiu( ulenucompatuENCODINGuencodinguadd_whitespaceutokensuappenduprev_rowuprev_coluNEWLINEuNLujoin(uselfuiterableututok_typeutokenustartuenduline((u%/mit/python/lib/python3.0/tokenize.pyu untokenizes    c Csd}g}|jj}|\}}|ttfko|d7}n|ttfko d}nd}x |D]} | dd\}}|tko||_ qon|ttfko|d7}n|t ko|od|}nd}nd}|t ko|j|qonb|t ko|j qonD|ttfko d}n'|o|o||dd}n||qoWdS(Nu iiFT(uFalseutokensuappenduNAMEuNUMBERuNEWLINEuNLuTrueuENCODINGuencodinguSTRINGuINDENTuDEDENTupop( uselfutokenuiterableu startlineuindentsu toks_appendutoknumutokvalu prevstringutok((u%/mit/python/lib/python3.0/tokenize.pyucompatsD             N(u__name__u __module__u__init__uadd_whitespaceu untokenizeucompat(u __locals__((u%/mit/python/lib/python3.0/tokenize.pyu Untokenizers    u UntokenizercCsBt}|j|}|jdk o|j|j}n|S(uTransform tokens back into Python source code. It returns a bytes object, encoded using the ENCODING token, which is the first token sequence output by tokenize. Each element returned by the iterable must be a token sequence with at least two elements, a token number and token value. If only two tokens are passed, the resulting output is poor. Round-trip invariant for full input: Untokenized source will match input source exactly Round-trip invariant for limited intput: # Output bytes will tokenize the back to the input t1 = [tok[:2] for tok in tokenize(f.readline)] newcode = untokenize(t1) readline = BytesIO(newcode).readline t2 = [tok[:2] for tok in tokenize(readline)] assert t1 == t2 N(u Untokenizeru untokenizeuencodinguNoneuencode(uiterableuutuout((u%/mit/python/lib/python3.0/tokenize.pyu untokenizes  csd}dd}fd}fd}|}|j|od|dd}n|p dgfS||}|o||gfS|}|pd|gfS||}|o|||gfSd||gfS( u The detect_encoding() function is used to detect the encoding that should be used to decode a Python source file. It requires one argment, readline, in the same way as the tokenize() generator. It will call readline a maximum of twice, and return the encoding used (as a string) and a list of any lines (left as bytes) it has read in. It detects the encoding from the presence of a utf-8 bom or an encoding cookie as specified in pep-0263. If both a bom and a cookie are present, but disagree, a SyntaxError will be raised. If no encoding is specified, then the default of 'utf-8' will be returned. scs*y SWntk o dSYnXdS(Ns(u StopIteration((ureadline(u%/mit/python/lib/python3.0/tokenize.pyu read_or_stops csy|jd}Wntk oYnTXtj|}|o<|d}o&t|jdkotdn|SdS(Nuasciiiuutf-8uencoding problem: utf-8(udecodeuUnicodeDecodeErroru cookie_reufindallulookupunameu SyntaxError(ulineu line_stringumatchesuencoding(u bom_found(u%/mit/python/lib/python3.0/tokenize.pyu find_cookie s iNuutf-8FT(uFalseuNoneu startswithuTrue(ureadlineuutf8_bomuencodingu read_or_stopu find_cookieufirstusecond((u bom_foundureadlineu%/mit/python/lib/python3.0/tokenize.pyudetect_encodings*     csCt\}}fd}t||}t|j|S(u The tokenize() generator requires one argment, readline, which must be a callable object which provides the same interface as the readline() method of built-in file objects. Each call to the function should return one line of input as bytes. Alternately, readline can be a callable function terminating with StopIteration: readline = open(myfile, 'rb').__next__ # Example of alternate readline The generator produces 5-tuples with these members: the token type; the token string; a 2-tuple (srow, scol) of ints specifying the row and column where the token begins in the source; a 2-tuple (erow, ecol) of ints specifying the row and column where the token ends in the source; and the line on which the token was found. The line passed is the logical line; continuation lines are included. The first token sequence will always be an ENCODING token which tells you which encoding was used to decode the bytes stream. c3s1x*y VWqtk o dSYqXqdS(N(u StopIteration((ureadline(u%/mit/python/lib/python3.0/tokenize.pyureadline_generatorCs  (udetect_encodinguchainu _tokenizeu__next__(ureadlineuencodinguconsumedureadline_generatoruchained((ureadlineu%/mit/python/lib/python3.0/tokenize.pyutokenize/sc cs d}}}tjdd}}d\}}d} dg} |dk ot|dd dfVnxHy |} Wntk od} YnX|dk o| j|} n|d}dt| } } |o | ptd|n|j| }|oU|j d} }t || d||||f| | fVd!\}}d} qx|oe| ddd koN| d dd ko7t || ||t| f| fVd}d} qeqx|| }| | } qen|dkof| o^| pPnd}x~| | kop| | d ko|d}nD| | d ko|t dt }n| | dko d}nP| d} q W| | koPn| | dko| | dko| | dj d}| t|}t||| f|| t|f| fVt| |d||f|t| f| fVqettf| | dk| | d|| f|t| f| fVqen|| dko;| j|t| d| |df|| f| fVnx|| dko`|| kotdd|| | fn| dd} td|| f|| f| fVqWn'| ptd|dfnd}x/| | ko!tj| | }|o|jd\}}||f||f|}}} | ||| |}}||kp'|dko2|dko%|dkot|||| fVq|dko,|dkotnt|||| fVq|dkot|||| fVq|tkot|}|j| | }|o=|j d} | || }t |||| f| fVqo||f}| |d}| } Pq|tkp.|ddtkp|ddtko|ddkoV||f}t|pt|dp t|d}| |dd}}| } Pqot |||| fVq||kot|||| fVq|dko d}q|dko|d}n|dko|d}nt|||| fVq{t | | || f|| df| fV| d} q{Wqex8| ddD]&}td|df|dfdfVqWtd|df|dfdfVdS("Niu_u 0123456789usiuEOF in multi-line stringiu\ iu\ u u u u# u#u iu3unindent does not match any outer indentation levelu uEOF in multi-line statementu.u...iiu u\u([{u)]}(ui(ii(ii(ui(ustringu ascii_lettersuNoneuENCODINGu StopIterationudecodeulenu TokenErrorumatchuenduSTRINGu ERRORTOKENutabsizeurstripuCOMMENTuNLuappenduINDENTuIndentationErroruDEDENTu pseudoproguspanuNUMBERuNEWLINEu triple_quoteduendprogsu single_quoteduNAMEuOPu ENDMARKER(ureadlineuencodingulnumuparenlevu continuedu namecharsunumcharsucontstruneedcontucontlineuindentsulineuposumaxustrstartuendproguendmatchuenducolumnu comment_tokenunl_posu pseudomatchustartusposueposutokenuinitialuindent((u%/mit/python/lib/python3.0/tokenize.pyu _tokenizeMs         5      $$$  . )   '             $cCs t|dS(N(u _tokenizeuNone(ureadline((u%/mit/python/lib/python3.0/tokenize.pyugenerate_tokenss(u'''u"""ur'''ur"""uR'''uR"""ub'''ub"""uB'''uB"""ubr'''ubr"""uBr'''uBr"""ubR'''ubR"""uBR'''uBR"""(u'u"ur'ur"uR'uR"ub'ub"uB'uB"ubr'ubr"uBr'uBr"ubR'ubR"uBR'uBR"(Ju__doc__u __author__u __credits__ureustringusysutokenucodecsulookupu itertoolsuchainurepeatucompileu cookie_reudiru__all__uN_TOKENSuCOMMENTutok_nameuNLuENCODINGugroupuanyumaybeu WhitespaceuCommentuIgnoreuNameu Hexnumberu Binnumberu Octnumberu Decnumberu IntnumberuExponentu PointfloatuExpfloatu Floatnumberu ImagnumberuNumberuSingleuDoubleuSingle3uDouble3uTripleuStringuOperatoruBracketuSpecialuFunnyu PlainTokenuTokenuContStru PseudoExtrasu PseudoTokenumapu tokenprogu pseudoprogu single3progu double3proguNoneuendprogsu triple_quotedutu single_quotedutabsizeu Exceptionu TokenErroruStopTokenizingu Untokenizeru untokenizeudetect_encodingutokenizeu _tokenizeugenerate_tokens(((u%/mit/python/lib/python3.0/tokenize.pyus$                '#J  =