Edit: /usr/lib64/python2.6/urllib2.pyc (45813B)
gc @ s d Z d d k Z d d k Z d d k Z d d k Z d d k Z d d k Z d d k Z d d k Z d d k Z d d k
Z
d d k Z d d k Z d d k
Z
y d d k l Z Wn# e j
o d d k l Z n Xd d k l Z l Z l Z l Z l Z l Z l Z l Z l Z l Z l Z l Z d d k l Z l Z l Z l! Z! e
i" d Z# d a% d e i& d Z' d Z( d e) f d
YZ* d e* e f d YZ+ e i, d
Z- d Z. d dC d YZ/ d dD d YZ0 d Z1 d dE d YZ2 d e2 f d YZ3 d e2 f d YZ4 d e2 f d YZ5 d Z6 d e2 f d YZ7 d dF d YZ8 d! e8 f d" YZ9 d# dG d$ YZ: d% e: e2 f d& YZ; d' e: e2 f d( YZ< d) Z= d* dH d+ YZ> d, e2 e> f d- YZ? d. e2 e> f d/ YZ@ d0 e2 f d1 YZA d2 eA f d3 YZB eC e d4 o d5 eA f d6 YZD n d7 e2 f d8 YZE d9 e2 f d: YZF d; ZG d< ZH d= e2 f d> YZI d? e2 f d@ YZJ dA eJ f dB YZK d S(I s! An extensible library for opening URLs using a variety of protocols
The simplest way to use this module is to call the urlopen function,
which accepts a string containing a URL or a Request object (described
below). It opens the URL and returns the results as file-like
object; the returned object has some extra methods described below.
The OpenerDirector manages a collection of Handler objects that do
all the actual work. Each Handler implements a particular protocol or
option. The OpenerDirector is a composite object that invokes the
Handlers needed to open the requested URL. For example, the
HTTPHandler performs HTTP GET and POST requests and deals with
non-error returns. The HTTPRedirectHandler automatically deals with
HTTP 301, 302, 303 and 307 redirect errors, and the HTTPDigestAuthHandler
deals with digest authentication.
urlopen(url, data=None) -- Basic usage is the same as original
urllib. pass the url and optionally data to post to an HTTP URL, and
get a file-like object back. One difference is that you can also pass
a Request instance instead of URL. Raises a URLError (subclass of
IOError); for HTTP errors, raises an HTTPError, which can also be
treated as a valid response.
build_opener -- Function that creates a new OpenerDirector instance.
Will install the default handlers. Accepts one or more Handlers as
arguments, either instances or Handler classes that it will
instantiate. If one of the argument is a subclass of the default
handler, the argument will be installed instead of the default.
install_opener -- Installs a new opener as the default opener.
objects of interest:
OpenerDirector -- Sets up the User Agent as the Python-urllib client and manages
the Handler classes, while dealing with requests and responses.
Request -- An object that encapsulates the state of a request. The
state can be as simple as the URL. It can also include extra HTTP
headers, e.g. a User-Agent.
BaseHandler --
exceptions:
URLError -- A subclass of IOError, individual protocols have their own
specific subclass.
HTTPError -- Also a valid HTTP response, so you can treat an HTTP error
as an exceptional event or valid response.
internals:
BaseHandler and parent
_call_chain conventions
Example usage:
import urllib2
# set up authentication info
authinfo = urllib2.HTTPBasicAuthHandler()
authinfo.add_password(realm='PDQ Application',
uri='https://mahler:8092/site-updates.py',
user='klem',
passwd='geheim$parole')
proxy_support = urllib2.ProxyHandler({"http" : "http://ahad-haam:3128"})
# build a new opener that adds authentication and caching FTP handlers
opener = urllib2.build_opener(proxy_support, authinfo, urllib2.CacheFTPHandler)
# install it
urllib2.install_opener(opener)
f = urllib2.urlopen('http://www.python.org/')
iN( t StringIO( t unwrapt unquotet splittypet splithostt quotet
addinfourlt splitportt splitattrt
ftpwrappert splitusert splitpasswdt
splitvalue( t localhostt url2pathnamet
getproxiest proxy_bypassi c C s- t d j o
t a n t i | | | S( N( t _openert Nonet build_openert open( t urlt datat timeout( ( s /usr/lib64/python2.6/urllib2.pyt urlopenz s
c C s
| a d S( N( R ( t opener( ( s /usr/lib64/python2.6/urllib2.pyt install_opener s t URLErrorc B s e Z d Z d Z RS( c C s | f | _ | | _ d S( N( t argst reason( t selfR ( ( s /usr/lib64/python2.6/urllib2.pyt __init__ s c C s d | i S( Ns
( R ( R ( ( s /usr/lib64/python2.6/urllib2.pyt __str__ s ( t __name__t
__module__R R ( ( ( s /usr/lib64/python2.6/urllib2.pyR s t HTTPErrorc B s) e Z d Z e i Z d Z d Z RS( sB Raised when HTTP error occurs, but also acts like non-error returnc C sX | | _ | | _ | | _ | | _ | | _ | d j o | i | | | | n d S( N( t codet msgt hdrst fpt filenameR t _HTTPError__super_init( R R R$ R% R& R' ( ( s /usr/lib64/python2.6/urllib2.pyR s
c C s d | i | i f S( Ns HTTP Error %s: %s( R$ R% ( R ( ( s /usr/lib64/python2.6/urllib2.pyR s ( R! R" t __doc__R R R) R ( ( ( s /usr/lib64/python2.6/urllib2.pyR# s
s :\d+$c C sa | i } t i | d } | d j o | i d d } n t i d | d } | i S( s Return request-host, as defined by RFC 2965.
Variation from RFC: returned value is lowercased, for convenient
comparison.
i t t Host( t get_full_urlt urlparset
get_headert _cut_port_ret subt lower( t requestR t host( ( s /usr/lib64/python2.6/urllib2.pyt request_host s
t Requestc B s e Z d h d e d Z d Z d Z d Z d Z d Z d Z
d Z d Z d Z
d
Z d Z d Z d
Z d Z d Z d Z d d Z d Z RS( c C s t | | _ d | _ d | _ d | _ d | _ | | _ h | _ x* | i D] \ } } | i
| | qR Wh | _ | d j o t | } n | | _
| | _ d S( N( R t _Request__originalR t typeR4 t portt _tunnel_hostR t headerst itemst
add_headert unredirected_hdrsR5 t origin_req_hostt unverifiable( R R R R; R? R@ t keyt value( ( s /usr/lib64/python2.6/urllib2.pyR s
c C sb | d d j oD | d } t t d | o" t | d | t | | Sn t | d S( Ni t _Request__r_t get_( t hasattrR6 t getattrt AttributeError( R t attrt name( ( s /usr/lib64/python2.6/urllib2.pyt __getattr__ s
c C s | i o d Sd Sd S( Nt POSTt GET( t has_data( R ( ( s /usr/lib64/python2.6/urllib2.pyt
get_method s
c C s
| | _ d S( N( R ( R R ( ( s /usr/lib64/python2.6/urllib2.pyt add_data s c C s
| i d j S( N( R R ( R ( ( s /usr/lib64/python2.6/urllib2.pyRM s c C s | i S( N( R ( R ( ( s /usr/lib64/python2.6/urllib2.pyt get_data s c C s | i S( N( R7 ( R ( ( s /usr/lib64/python2.6/urllib2.pyR- s c C sZ | i d j oC t | i \ | _ | _ | i d j o t d | i qS n | i S( Ns unknown url type: %s( R8 R R R7 t _Request__r_typet
ValueError( R ( ( s /usr/lib64/python2.6/urllib2.pyt get_type s
c C sV | i d j o? t | i \ | _ | _ | i o t | i | _ qO n | i S( N( R4 R R RQ t _Request__r_hostR ( R ( ( s /usr/lib64/python2.6/urllib2.pyt get_host s
c C s | i S( N( RT ( R ( ( s /usr/lib64/python2.6/urllib2.pyt get_selector s c C sM | i d j o | i o | i | _ n | | _ | i | _ | | _ d S( Nt https( R8 R: R4 R7 RT ( R R4 R8 ( ( s /usr/lib64/python2.6/urllib2.pyt set_proxy s
c C s | i | i j S( N( RT R7 ( R ( ( s /usr/lib64/python2.6/urllib2.pyt has_proxy
s c C s | i S( N( R? ( R ( ( s /usr/lib64/python2.6/urllib2.pyt get_origin_req_host
s c C s | i S( N( R@ ( R ( ( s /usr/lib64/python2.6/urllib2.pyt is_unverifiable s c C s | | i | i R\ ( R RA R] ( ( s /usr/lib64/python2.6/urllib2.pyt add_unredirected_header s c C s | | i j p
| | i j S( N( R; R> ( R t header_name( ( s /usr/lib64/python2.6/urllib2.pyt
has_header s c C s" | i i | | i i | | S( N( R; t getR> ( R R_ t default( ( s /usr/lib64/python2.6/urllib2.pyR/ s c C s) | i i } | i | i | i S( N( R> t copyt updateR; R< ( R R& ( ( s /usr/lib64/python2.6/urllib2.pyt header_items$ s N( R! R" R t FalseR RJ RN RO RM RP R- RS RU RV RX RY RZ R[ R= R^ R` R/ Re ( ( ( s /usr/lib64/python2.6/urllib2.pyR6 s( t OpenerDirectorc B sS e Z d Z d Z d Z d Z d e i d Z d d Z
d Z RS( c C sM d t } d | f g | _ g | _ h | _ h | _ h | _ h | _ d S( Ns Python-urllib/%ss
User-agent( t __version__t
addheaderst handlerst handle_opent handle_errort process_responset process_request( R t client_version( ( s /usr/lib64/python2.6/urllib2.pyR * s
c
C s t | d p t d t | n t } xst | D]e} | d j o q= n | i d } | | } | | d } | i d oq | i d | d } | | d } y t | } Wn t j
o n X| i i
| h } | | i | $ s R; R? R@ N( i- i. i/ i3 ( s GETR ( i- i. i/ (
RN t replaceR R; R< R6 RZ R R# R- ( R R R' R$ R% R; t newurlt mt
newheaders( ( s /usr/lib64/python2.6/urllib2.pyRq s
c C s d | j o | i d d } n) d | j o | i d d } n d St i | } | i p t | } d | d >> _parse_proxy('file:/ftp.example.com/')
Traceback (most recent call last):
ValueError: proxy URL with no authority: 'file:/ftp.example.com/'
The first three items of the returned tuple may be None.
Examples of authority parsing:
>>> _parse_proxy('proxy.example.com')
(None, None, None, 'proxy.example.com')
>>> _parse_proxy('proxy.example.com:3128')
(None, None, None, 'proxy.example.com:3128')
The authority component may optionally include userinfo (assumed to be
username:password):
>>> _parse_proxy('joe:password@proxy.example.com')
(None, 'joe', 'password', 'proxy.example.com')
>>> _parse_proxy('joe:password@proxy.example.com:3128')
(None, 'joe', 'password', 'proxy.example.com:3128')
Same examples, but with URLs instead:
>>> _parse_proxy('http://proxy.example.com/')
('http', None, None, 'proxy.example.com')
>>> _parse_proxy('http://proxy.example.com:3128/')
('http', None, None, 'proxy.example.com:3128')
>>> _parse_proxy('http://joe:password@proxy.example.com/')
('http', 'joe', 'password', 'proxy.example.com')
>>> _parse_proxy('http://joe:password@proxy.example.com:3128')
('http', 'joe', 'password', 'proxy.example.com:3128')
Everything after the authority is ignored:
>>> _parse_proxy('ftp://joe:password@proxy.example.com/rubbish:3128')
('ftp', 'joe', 'password', 'proxy.example.com')
Test for no trailing '/' case:
>>> _parse_proxy('http://joe:password@proxy.example.com')
('http', 'joe', 'password', 'proxy.example.com')
R s //s proxy URL with no authority: %ri iN( R Rz R RR Ry R
R ( t proxyt schemet r_schemet authorityt endt userinfot hostportt usert password( ( s /usr/lib64/python2.6/urllib2.pyt _parse_proxyq s 2
R c B s# e Z d Z d d Z d Z RS( id c C s | d j o
t } n t | d p
t d | | _ x= | i D]/ \ } } t | d | | | | i d qJ Wd S( Nt has_keys proxies must be a mappings %s_openc S s | | | | S( ( ( t rR R8 R ( ( s /usr/lib64/python2.6/urllib2.pyt s ( R R RE t AssertionErrort proxiesR< t setattrRs ( R R R8 R ( ( s /usr/lib64/python2.6/urllib2.pyR s
c C s | i } t | \ } } } } | d j o
| } n | i | i o t | i o d S| oP | oI d t | t | f } t i | i }
| i
d d |
n t | } | i | | | | j p
| d j o d S| i i
| d | i Sd S( Ns %s:%ss Proxy-authorizations Basic RW R ( RS R R RU R4 R R t base64t b64encodet stripR= RX R R R ( R R R R8 t orig_typet
proxy_typeR R R t user_passt creds( ( s /usr/lib64/python2.6/urllib2.pyRs s
N( R! R" R R R Rs ( ( ( s /usr/lib64/python2.6/urllib2.pyR s
t HTTPPasswordMgrc B s8 e Z d Z d Z d Z e d Z d Z RS( c C s
h | _ d S( N( t passwd( R ( ( s /usr/lib64/python2.6/urllib2.pyR s c C s t | t o
| g } n | | i j o h | i | username="%s", realm="%s", nonce="%s", uri="%s", response="%s"s
, opaque="%s"s
, digest="%s"s , algorithm="%s"s , qop=auth, nc=%s, cnonce="%s"( Ra R t KeyErrort get_algorithm_implsR R
R- RM t get_entity_digestRP RN RV R= R< RP R ( R R RF R RN RQ RR RT t Ht KDR R$ t entdigt A1t A2t ncvaluet cnoncet noncebitt respdigR ( ( s /usr/lib64/python2.6/urllib2.pyRC sV
!
)
c sY | i } | d j o
d n | d j o
d n f d } | f S( NRS c S s t i | i S( ( RI t md5RM ( t x( ( s /usr/lib64/python2.6/urllib2.pyR s t SHAc S s t i | i S( ( RI RJ RM ( Rb ( ( s /usr/lib64/python2.6/urllib2.pyR s c s d | | f S( s %s:%s( ( R8 t d( RX ( s /usr/lib64/python2.6/urllib2.pyR s ( t upper( R RR RY ( ( RX s /usr/lib64/python2.6/urllib2.pyRV s
c C s d S( N( R ( R R RF ( ( s /usr/lib64/python2.6/urllib2.pyRW ! s N( R! R" R R R R" R@ RP RC RV RW ( ( ( s /usr/lib64/python2.6/urllib2.pyR; s
= t HTTPDigestAuthHandlerc B s# e Z d Z d Z d Z d Z RS( s An authentication protocol defined by RFC 2069
Digest authentication improves on basic authentication because it
does not transmit passwords in the clear.
R+ i c C s? t i | i d } | i d | | | } | i | S( Ni s www-authenticate( R. R- R" R ( R R R' R$ R% R; R4 t retry( ( s /usr/lib64/python2.6/urllib2.pyR, 0 s
( R! R" R* R# R R, ( ( ( s /usr/lib64/python2.6/urllib2.pyRf & s t ProxyDigestAuthHandlerc B s e Z d Z d Z d Z RS( s Proxy-Authorizationi c C s2 | i } | i d | | | } | i | S( Ns proxy-authenticate( RU R" R ( R R R' R$ R% R; R4 Rg ( ( s /usr/lib64/python2.6/urllib2.pyR. = s
( R! R" R# R R. ( ( ( s /usr/lib64/python2.6/urllib2.pyRh 8 s t AbstractHTTPHandlerc B s/ e Z d d Z d Z d Z d Z RS( i c C s
| | _ d S( N( t _debuglevel( R t
debuglevel( ( s /usr/lib64/python2.6/urllib2.pyR F s c C s
| | _ d S( N( Rj ( R t level( ( s /usr/lib64/python2.6/urllib2.pyt set_http_debuglevelI s c
C sH | i } | p t d n | i ob | i } | i d p | i d d n | i d p | i d d t | q n | } | i o. t | i \ } } t
| \ } } n | i d p | i d | n xJ | i i D]< \ } } | i
} | i | p | i | | qqW| S( Ns
no host givens Content-types! application/x-www-form-urlencodeds Content-lengths %dR, ( RU R RM RP R` R^ R RY R RV R R Ri R\ (
R R3 R4 R t sel_hostR t selt sel_pathRI RB ( ( s /usr/lib64/python2.6/urllib2.pyt do_request_L s0
c C s | i } | p t d n | | d | i } | i | i t | i } | i | i d | d s s Proxy-AuthorizationR; ( RU R R t set_debuglevelRj R R; Rd R> R< R: t _set_tunnelR3 RN RV R t getresponseR Ru R t recvt _fileobjectR R R% R- t statusR$ R ( R t
http_classR R4 R R; t tunnel_headerst proxy_auth_hdrR t errR' RH ( ( s /usr/lib64/python2.6/urllib2.pyRr i s8
%
( R! R" R Rm Rq Rr ( ( ( s /usr/lib64/python2.6/urllib2.pyRi D s R c B s e Z d Z e i Z RS( c C s | i t i | S( N( Rr R t HTTPConnection( R R ( ( s /usr/lib64/python2.6/urllib2.pyt http_open s ( R! R" R Ri Rq t http_request( ( ( s /usr/lib64/python2.6/urllib2.pyR s R R c B s e Z d Z e i Z RS( c C s | i t i | S( N( Rr R t HTTPSConnection( R R ( ( s /usr/lib64/python2.6/urllib2.pyt
https_open s ( R! R" R Ri Rq t
https_request( ( ( s /usr/lib64/python2.6/urllib2.pyR s t HTTPCookieProcessorc B s2 e Z d d Z d Z d Z e Z e Z RS( c C s6 d d k } | d j o | i } n | | _ d S( Ni( t cookielibR t CookieJart cookiejar( R R R ( ( s /usr/lib64/python2.6/urllib2.pyR s
c C s | i i | | S( N( R t add_cookie_header( R R3 ( ( s /usr/lib64/python2.6/urllib2.pyR s c C s | i i | | | S( N( R t extract_cookies( R R3 Rv ( ( s /usr/lib64/python2.6/urllib2.pyR s N( R! R" R R R R R R ( ( ( s /usr/lib64/python2.6/urllib2.pyR s
R c B s e Z d Z RS( c C s | i } t d | d S( Ns unknown url type: %s( RS R ( R R R8 ( ( s /usr/lib64/python2.6/urllib2.pyR s ( R! R" R ( ( ( s /usr/lib64/python2.6/urllib2.pyR s c C sp h } xc | D][ } | i d d \ } } | d d j o" | d d j o | d d !} n | | | Parse list of key=value strings where keys are not duplicated.t =i i t "i( R? ( t lt parsedt eltR R ( ( s /usr/lib64/python2.6/urllib2.pyRA s "c C s g } d } t } } x | D] } | o | | 7} t } q n | oB | d j o
t } q n | d j o
t } n | | 7} q n | d j o | i | d } q n | d j o
t } n | | 7} q W| o | i | n g } | D] } | | i q ~ S( sp Parse lists as described by RFC 2068 Section 2.
In particular, parse comma-separated lists where the elements of
the list may include quoted-strings. A quoted-string could
contain a comma. A non-quoted string could have quotes in the
middle. Neither commas nor quotes count if they are escaped.
Only double-quotes count, not single-quotes.
R+ s \R t ,( Rf R R R ( R8 t rest partt escapeR t curR ( ( s /usr/lib64/python2.6/urllib2.pyRB s6
R c B s) e Z d Z d Z d Z d Z RS( c C sv | i } | d d j oH | d d !d j o4 | i o* | i d j o d | _ | i i | S| i | Sd S( Ni s //i R R
t ftp( RV R4 R8 R R t open_local_file( R R R ( ( s /usr/lib64/python2.6/urllib2.pyt file_open s / c C s t i d j oi y7 t t i d d t i t i d t _ Wqy t i j
o t i d f t _ qy Xn t i S( NR
i ( R t namesR R R t gethostbyname_ext gethostnamet gaierrort
gethostbyname( R ( ( s /usr/lib64/python2.6/urllib2.pyt get_names s $ c C si d d k } d d k } | i } | i } t | } y t i | } | i } | i i | i
d t } | i | d }
t
i t d |
p d | | f } | o t | \ } } n | p$ | oY t i | | i j o= | o d | | }
n d | }
t t | d | |
SWn! t j
o } t | n Xt d d S( Nit usegmti s6 Content-type: %s
Content-length: %d
Last-modified: %s
s
text/plains file://t rbs file not on local host( t email.utilst mimetypesRU RV R R/ t statt st_sizet utilst
formatdatet st_mtimeR t
guess_typet mimetoolst MessageR R R R R R R t OSErrorR ( R R t emailR R4 R( t localfilet statst sizet modifiedt mtypeR; R9 t origurlR% ( ( s /usr/lib64/python2.6/urllib2.pyR s0 $
N( R! R" R R R R R ( ( ( s /usr/lib64/python2.6/urllib2.pyR s
R c B s e Z d Z d Z RS( c C s d d k } d d k } | i } | p t d n t | \ } } | d j o
| i } n
t | } t | \ } } | o t | \ } } n d } t
| } t
| p d } t
| p d } y t i | } Wn$ t i
j
o } t | n Xt | i \ } }
| i d } t t
| } | d | d } } | o | d o | d } n y9| i | | | | | | i }
| o d p d } xP |
D]H } t | \ } } | i d j o | d j o | i } qqW|
i | | \ } } d } | i | i d } | o | d | 7} n | d j o | d j o | d | 7} n t | } t i | } t | | | i SWn2 | i j
o# } t d | t i d n Xd S( Nis ftp error: no host givenR+ R i i R) t DR8 t at AR Rd s Content-type: %s
s Content-length: %d
s
ftp error: %si ( R R R R) Rd R (! t ftplibR RU R R R t FTP_PORTR{ R
R R R R Ru R RV R? t mapt connect_ftpR R R2 Re t retrfileR R- R R R R t
all_errorst syst exc_info( R R R R R4 R9 R R R% R t attrst dirst filet fwR8 RH RB R' t retrlenR; R t sf( ( s /usr/lib64/python2.6/urllib2.pyt ftp_open: s^
!
c C s t | | | | | | } | S( N( R ( R R R R4 R9 R R R ( ( s /usr/lib64/python2.6/urllib2.pyR o s ( R! R" R R ( ( ( s /usr/lib64/python2.6/urllib2.pyR 9 s 5t CacheFTPHandlerc B s5 e Z d Z d Z d Z d Z d Z RS( c C s1 h | _ h | _ d | _ d | _ d | _ d S( Ni i< i ( t cacheR t soonestt delayt max_conns( R ( ( s /usr/lib64/python2.6/urllib2.pyR w s
c C s
| | _ d S( N( R ( R t t( ( s /usr/lib64/python2.6/urllib2.pyt
setTimeout~ s c C s
| | _ d S( N( R ( R R ( ( s /usr/lib64/python2.6/urllib2.pyt setMaxConns s c C s | | | d i | | f } | | i j o t i | i | i | L sp R"
n 'i H-@
@ f
+4;