/usr/lib/python2.6/site-packages/concurrent/futures
Edit: /usr/lib/python2.6/site-packages/concurrent/futures/process.pyc (12642B)
Ñò
m
Vc
@ s! d Z d d k Z d d k l Z d d k Z d d k Z d d k Z d d k Z d d k Z d Z
e i ƒ Z e
a d „ Z d Z d e f d „ ƒ YZ d e f d
„ ƒ YZ d e f d „ ƒ YZ d
„ Z d „ Z d „ Z e
a d a d „ Z d e i f d „ ƒ YZ e i e ƒ d S( s+ Implements ProcessPoolExecutor.
The follow diagram and text describe the data-flow through the system:
|======================= In-process =====================|== Out-of-process ==|
+----------+ +----------+ +--------+ +-----------+ +---------+
| | => | Work Ids | => | | => | Call Q | => | |
| | +----------+ | | +-----------+ | |
| | | ... | | | | ... | | |
| | | 6 | | | | 5, call() | | |
| | | 7 | | | | ... | | |
| Process | | ... | | Local | +-----------+ | Process |
| Pool | +----------+ | Worker | | #1..n |
| Executor | | Thread | | |
| | +----------- + | | +-----------+ | |
| | <=> | Work Items | <=> | | <= | Result Q | <= | |
| | +------------+ | | +-----------+ | |
| | | 6: call() | | | | ... | | |
| | | future | | | | 4, result | | |
| | | ... | | | | 3, except | | |
+----------+ +------------+ +--------+ +-----------+ +---------+
Executor.submit() called:
- creates a uniquely numbered _WorkItem and adds it to the "Work Items" dict
- adds the id of the _WorkItem to the "Work Ids" queue
Local worker thread:
- reads work ids from the "Work Ids" queue and looks up the corresponding
WorkItem from the "Work Items" dict: if the work item has been cancelled then
it is simply removed from the dict, otherwise it is repackaged as a
_CallItem and put in the "Call Q". New _CallItems are put in the "Call Q"
until "Call Q" is full. NOTE: the size of the "Call Q" is kept small because
calls placed in the "Call Q" can no longer be cancelled with Future.cancel().
- reads _ResultItems from "Result Q", updates the future stored in the
"Work Items" dict and deletes the dict entry
Process #1..n:
- reads _CallItems from "Call Q", executes the calls, and puts the resulting
_ResultItems in "Request Q"
iÿÿÿÿN( t _bases" Brian Quinlan (brian@sweetapp.com)c C so t a t o t t i ƒ ƒ n d } x! | D] \ } } | i d ƒ q- Wx | D] \ } } | i ƒ qQ Wd S( N( ( t Truet _shutdownt _threads_queuest listt itemst putt Nonet join( R t tt q( ( s; /tmp/pip-build-U5v4wA/futures/concurrent/futures/process.pyt _python_exitI s i t _WorkItemc B s e Z d „ Z RS( c C s( | | _ | | _ | | _ | | _ d S( N( t futuret fnt argst kwargs( t selfR
R R R ( ( s; /tmp/pip-build-U5v4wA/futures/concurrent/futures/process.pyt __init__Y s ( t __name__t
__module__R ( ( ( s; /tmp/pip-build-U5v4wA/futures/concurrent/futures/process.pyR X s t _ResultItemc B s e Z d d d „ Z RS( c C s | | _ | | _ | | _ d S( N( t work_idt exceptiont result( R R R R ( ( s; /tmp/pip-build-U5v4wA/futures/concurrent/futures/process.pyR ` s N( R R R R ( ( ( s; /tmp/pip-build-U5v4wA/futures/concurrent/futures/process.pyR _ s t _CallItemc B s e Z d „ Z RS( c C s( | | _ | | _ | | _ | | _ d S( N( R R R R ( R R R R R ( ( s; /tmp/pip-build-U5v4wA/futures/concurrent/futures/process.pyR f s ( R R R ( ( ( s; /tmp/pip-build-U5v4wA/futures/concurrent/futures/process.pyR e s c C s¾ x· t o¯ | i d t ƒ } | d j o | i d ƒ d Sy | i | i | i Ž } Wn? t j
o3 t i ƒ d } | i t
| i d | ƒƒ q X| i t
| i d | ƒƒ q Wd S( sø Evaluates calls from call_queue and places the results in result_queue.
This worker is run in a separate process.
Args:
call_queue: A multiprocessing.Queue of _CallItems that will be read and
evaluated by the worker.
result_queue: A multiprocessing.Queue of _ResultItems that will written
to by the worker.
shutdown: A multiprocessing.Event that will be set as a signal to the
worker that it should exit when call_queue is empty.
t blockNi R R ( R t getR R R R R t
BaseExceptiont syst exc_infoR R ( t
call_queuet result_queuet call_itemt rt e( ( s; /tmp/pip-build-U5v4wA/futures/concurrent/futures/process.pyt _process_workerl s
c C s¨ x¡ t o™ | i ƒ o d Sy | i d t ƒ } Wn t i j
o d SX| | } | i i ƒ o/ | i t | | i
| i | i ƒ d t ƒq | | =q q Wd S( sM Fills call_queue with _WorkItems from pending_work_items.
This function never blocks.
Args:
pending_work_items: A dict mapping work ids to _WorkItems e.g.
{5: <_WorkItem...>, 6: <_WorkItem...>, ...}
work_ids: A queue.Queue of work ids e.g. Queue([5, 6, ...]). Work ids
are consumed and the corresponding _WorkItems from
pending_work_items are transformed into _CallItems and put in
call_queue.
call_queue: A multiprocessing.Queue that will be filled with _CallItems
derived from _WorkItems.
NR (
R t fullR t Falset queuet EmptyR
t set_running_or_notify_cancelR R R R R ( t pending_work_itemst work_idsR R t work_item( ( s; /tmp/pip-build-U5v4wA/futures/concurrent/futures/process.pyt _add_call_item_to_queue‰ s"
c s4 d g ‰ ‡ ‡ f d † } xt o
t | | ˆ ƒ | i d t ƒ } | d j oR | | i } | | i =| i o | i i | i ƒ n | i i | i ƒ ~ n | ƒ } t
p | d j p
| i o[ | pP x# ˆ d t | ƒ j o | ƒ q× Wx | D] }
|
i
ƒ qWˆ i ƒ d Sn ~ q Wd S( s‘ Manages the communication between this process and the worker processes.
This function is run in a local thread.
Args:
executor_reference: A weakref.ref to the ProcessPoolExecutor that owns
this thread. Used to determine if the ProcessPoolExecutor has been
garbage collected and that this function can exit.
process: A list of the multiprocessing.Process instances used as
workers.
pending_work_items: A dict mapping work ids to _WorkItems e.g.
{5: <_WorkItem...>, 6: <_WorkItem...>, ...}
work_ids_queue: A queue.Queue of work ids e.g. Queue([5, 6, ...]).
call_queue: A multiprocessing.Queue that will be filled with _CallItems
derived from _WorkItems for processing by the process workers.
result_queue: A multiprocessing.Queue of _ResultItems generated by the
process workers.
i c s! ˆ i d ƒ ˆ d c d 7
RE Rn RF t register( ( ( s; /tmp/pip-build-U5v4wA/futures/concurrent/futures/process.pyt , s.
% B ^