<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <html> <head> <title>[execnet] </title> <meta content="text/html;charset=ISO-8859-1" name="Content-Type"/> <link href="style.css" media="screen" rel="stylesheet" type="text/css"/></head> <body> <div><a href="http://codespeak.net"><img alt="py lib" height="114" id="pyimg" src="http://codespeak.net/img/pylib.png" width="154"/></a></div> <div id="metaspace"> <div class="project_title">[execnet] </div> <div id="menubar"><a class="menu" href="index.html">index</a> <a class="menu" href="../../apigen/api/index.html">api</a> <a class="menu" href="../../apigen/source/index.html">source</a> <a class="menu" href="contact.html">contact</a> <a class="menu" href="download.html">download</a></div></div> <div id="contentspace"> <div id="docinfoline"> <div style="float: right; font-style: italic;"> </div></div> <div class="section" id="the-py-execnet-library"> <h1><a class="toc-backref" href="#id3">1 The py.execnet library</a></h1> <div class="contents topic" id="contents"> <p class="topic-title first">Contents</p> <ul class="auto-toc simple"> <li><a class="reference internal" href="#the-py-execnet-library" id="id3">1 The py.execnet library</a><ul class="auto-toc"> <li><a class="reference internal" href="#a-new-view-on-distributed-execution" id="id4">1.1 A new view on distributed execution</a></li> <li><a class="reference internal" href="#what-about-security-are-you-completely-nuts" id="id5">1.2 What about Security? Are you completely nuts?</a></li> </ul> </li> <li><a class="reference internal" href="#basic-features" id="id6">2 Basic Features</a><ul class="auto-toc"> <li><a class="reference internal" href="#available-gateways-connection-methods" id="id7">2.1 Available Gateways/Connection methods</a></li> <li><a class="reference internal" href="#remote-execution-approach" id="id8">2.2 Remote execution approach</a></li> <li><a class="reference internal" href="#the-channel-interface-for-exchanging-data-across-gateways" id="id9">2.3 The <strong>Channel</strong> interface for exchanging data across gateways</a><ul class="auto-toc"> <li><a class="reference internal" href="#the-complete-fileserver-example" id="id10">2.3.1 The complete Fileserver example</a></li> <li><a class="reference internal" href="#a-more-complicated-nested-gateway-example" id="id11">2.3.2 A more complicated "nested" Gateway Example</a></li> </ul> </li> </ul> </li> </ul> </div> <div class="section" id="a-new-view-on-distributed-execution"> <h2><a class="toc-backref" href="#id4">1.1 A new view on distributed execution</a></h2> <p><tt class="docutils literal"><span class="pre">py.execnet</span></tt> supports ad-hoc distribution of parts of a program across process and network barriers. <em>Ad-hoc</em> means that the client side may completely control</p> <ul class="simple"> <li>which parts of a program execute remotely and</li> <li>which data protocols are used between them</li> </ul> <p>without requiring any prior manual installation of user program code on the remote side. In fact, not even a prior installation of any server code is required, provided there is a way to get an input/output connection to a python interpreter (for example via "ssh" and a "python" executable).</p> <p>By comparison, traditional Remote Method Based (RMI) require prior installation and manual rather heavy processes of setup, distribution and communication between program parts.</p> </div> <div class="section" id="what-about-security-are-you-completely-nuts"> <h2><a class="toc-backref" href="#id5">1.2 What about Security? Are you completely nuts?</a></h2> <p>We'll talk about that later :-)</p> </div> </div> <div class="section" id="basic-features"> <h1><a class="toc-backref" href="#id6">2 Basic Features</a></h1> <p>With ''py.execnet'' you get the means</p> <ul class="simple"> <li>to navigate through the network with Process, Thread, SSH and Socket- gateways that allow you ...</li> <li>to distribute your program across a network and define communication protocols from the client side, making server maintenance superflous. In fact, there is no such thing as a server. It's just another computer ... if it doesn't run in a kernel-level jail <a class="footnote-reference" href="#id2" id="id1">[1]</a> in which case even that is virtualized.</li> </ul> <div class="section" id="available-gateways-connection-methods"> <h2><a class="toc-backref" href="#id7">2.1 Available Gateways/Connection methods</a></h2> <p>You may use one of the following connection methods:</p> <ul class="simple"> <li><a class="reference external" href="../../apigen/api/execnet.PopenGateway.html">py.execnet.PopenGateway</a> a subprocess on the local machine. Useful for jailing certain parts of a program or for making use of multiple processors.</li> <li><a class="reference external" href="../../apigen/api/execnet.SshGateway.html">py.execnet.SshGateway</a> a way to connect to a remote ssh server and distribute execution to it.</li> <li><a class="reference external" href="../../apigen/api/execnet.SocketGateway.html">py.execnet.SocketGateway</a> a way to connect to a remote Socket based server. <em>Note</em> that this method requires a manually started :source:py/execnet/script/socketserver.py script. You can run this "server script" without having the py lib installed on that remote system.</li> </ul> </div> <div class="section" id="remote-execution-approach"> <h2><a class="toc-backref" href="#id8">2.2 Remote execution approach</a></h2> <p>All gateways offer one main high level function:</p> <blockquote> <dl class="docutils"> <dt>def remote_exec(source):</dt> <dd>"""return channel object for communicating with the asynchronously executing 'source' code which will have a corresponding 'channel' object in its executing namespace."""</dd> </dl> </blockquote> <p>With <span class="incremental">remote_exec</span> you send source code to the other side and get both a local and a remote <a class="reference internal" href="#channel">Channel</a> object, which you can use to have the local and remote site communicate data in a structured way. Here is an example:</p> <blockquote> <pre class="doctest-block"> >>> import py >>> gw = py.execnet.PopenGateway() >>> channel = gw.remote_exec(""" ... import os ... channel.send(os.getpid()) ... """) >>> remote_pid = channel.receive() >>> remote_pid != py.std.os.getpid() True </pre> </blockquote> <p><span class="incremental">remote_exec</span> implements the idea to <tt class="docutils literal"><span class="pre">determine</span> <span class="pre">protocol</span> <span class="pre">and</span> <span class="pre">remote</span> <span class="pre">code</span> <span class="pre">from</span> <span class="pre">the</span> <span class="pre">client/local</span> <span class="pre">side</span></tt>. This makes distributing a program run in an ad-hoc manner (using e.g. <a class="reference external" href="../../apigen/api/execnet.SshGateway.html">py.execnet.SshGateway</a>) very easy.</p> <p>You should not need to maintain software on the other sides you are running your code at, other than the Python executable itself.</p> </div> <div class="section" id="the-channel-interface-for-exchanging-data-across-gateways"> <span id="exchange-data"></span><span id="channel-api"></span><span id="channel"></span><h2><a class="toc-backref" href="#id9">2.3 The <strong>Channel</strong> interface for exchanging data across gateways</a></h2> <p>While executing custom strings on "the other side" is simple enough it is often tricky to deal with. Therefore we want a way to send data items to and fro between the distributedly running program. The idea is to inject a Channel object for each execution of source code. This Channel object allows two program parts to send data to each other. Here is the current interface:</p> <pre class="literal-block"> # # API for sending and receiving anonymous values # channel.send(item): sends the given item to the other side of the channel, possibly blocking if the sender queue is full. Note that items need to be marshallable (all basic python types are): channel.receive(): receives an item that was sent from the other side, possibly blocking if there is none. Note that exceptions from the other side will be reraised as gateway.RemoteError exceptions containing a textual representation of the remote traceback. channel.waitclose(timeout=None): wait until this channel is closed. Note that a closed channel may still hold items that will be received or send. Note that exceptions from the other side will be reraised as gateway.RemoteError exceptions containing a textual representation of the remote traceback. channel.close(): close this channel on both the local and the remote side. A remote side blocking on receive() on this channel will get woken up and see an EOFError exception. </pre> <div class="section" id="the-complete-fileserver-example"> <h3><a class="toc-backref" href="#id10">2.3.1 The complete Fileserver example</a></h3> <p>problem: retrieving contents of remote files:</p> <pre class="literal-block"> import py contentserverbootstrap = py.code.Source( """ for fn in channel: f = open(fn, 'rb') try: channel.send(f.read()) finally: f.close() """) # open a gateway to a fresh child process contentgateway = py.execnet.SshGateway('codespeak.net') channel = contentgateway.remote_exec(contentserverbootstrap) for fn in somefilelist: channel.send(fn) content = channel.receive() # process content # later you can exit / close down the gateway contentgateway.exit() </pre> </div> <div class="section" id="a-more-complicated-nested-gateway-example"> <h3><a class="toc-backref" href="#id11">2.3.2 A more complicated "nested" Gateway Example</a></h3> <p>The following example opens a PopenGateway, i.e. a python child process, starts a socket server within that process and then opens a SocketGateway to the freshly started socketserver. Thus it forms a "triangle":</p> <pre class="literal-block"> CLIENT < ... > PopenGateway() < . . . . . . . > SocketGateway() </pre> <p>The below "socketserver" mentioned script is a small script that basically listens and accepts socket connections, receives one liners and executes them.</p> <p>Here are 20 lines of code making the above triangle happen:</p> <pre class="literal-block"> import py port = 7770 socketserverbootstrap = py.code.Source( mypath.dirpath().dirpath('bin', 'socketserver.py').read(), """ import socket sock = bind_and_listen(("localhost", %r)) channel.send("ok") startserver(sock) """ % port) # open a gateway to a fresh child process proxygw = py.execnet.PopenGateway() # execute asynchronously the above socketserverbootstrap on the other channel = proxygw.remote_exec(socketserverbootstrap) # the other side should start the socket server now assert channel.receive() == "ok" gw = py.execnet.SocketGateway('localhost', cls.port) print "initialized socket gateway to port", cls.port </pre> <table class="docutils footnote" frame="void" id="id2" rules="none"> <colgroup><col class="label" /><col /></colgroup> <tbody valign="top"> <tr><td class="label"><a class="fn-backref" href="#id1">[1]</a></td><td>There is an interesting emerging <a class="reference external" href="http://books.rsbac.org/unstable/x2223.html">Jail</a> linux technology as well as a host of others, of course.</td></tr> </tbody> </table> </div> </div> </div> </div></body></html>