remoteD

Document Version: 0.8, 12/22/2003

1. General Information

1.1 Introduction:

     remoteD is an easy to use interface for forked multi-process programming. It provides a simple dictionary interface for accessing data shared between processes and is capable of auto starting a server process when needed. Any python object that can be pickled can be stored in a remoteD 'share'. The name comes from the idea of a remote Dictionary object, which is the interface provided to your program. reads and writes are atomic, and explicit locking is available for situations that call for multi-command consistency.

1.2 Installation

remoteD is installed via the distutils package using the normal python setup.py mechanism. The full instructions for installing remoteD are:

1.3 License:

remoteD is currently distributed under a (new) BSD style license. to view the full text of this license click here.  Commercial support and customization of remoteD is available, please contact NeuroKode Labs, LLC for more information.

2 Objects, methods and constants

Constants:

UNIXSOCK - used to represent a UNIX socket type

IPSOCK - used to represent a IP socket type

DEFAULTSOCK - normally IPSOCK

Method/Attribute: Description:
2.1 Module Level Functions
initShare()

Summary: initShare returns a stubShare object in the calling process, and will fork and start a local share-server if one does not already exist.

Usage: SharedD = remoteD.initShare()

Arguments:

  • port: the port to connect to, or create the server on.
  • sType: Socket Type (UNIXSOCK, IPSOCK, DEFAULTSOCK)
  • AutoKill: whether the server should kill itself if 5 seconds pass with no children, normally 1 (yes).
createShareServer()

(Internal) createShareServer() is used by initShare() to create and fork a server.
do_command()

(Internal) do_command() is used by stubShare to preform commands on the share object.
2.2 stubShare Objects
stubShare objects behave like dictionary objects in most respects.
__init__()

Summary: creates a stubShare

Usage: myShare = stubShare(port, sType, host)

Arguments:

  • port - the port to connect to, a number for sType == IPSOCK
  • sType - the type of socket, can be IPSOCK or UNIXSOCK
  • host - the hostname to connect to.

  • Note: stubShare objects are normally created for you when a call to initShare is made, or from a call to newProc on an existing stubShare object. You should only need to create your own stubShare object if you are connecting to a remoteD share-server on a remote host.
has_key() Just like a dictionary object, but checks for the key's existence in the remoteD share-server
keys() Just like a dictionary object, but retrieves keys from the server. You should assign the results from this and reuse when possible to keep from pounding the server - see section 3.
Lock()

Summary: Locks the server associated with the share object

Usage: SharedD.Lock()

Unlock()

Summary: Unlocks a locked share object

Usage: SharedD.Unlock()

newProc()

Summary: stubShare.newProc() forks then performs the function passed to it.

Usage: SharedD.newProc(child_function, [arg1, arg2])

Note: The child_function must expect a stubShare object as it's first argument. This new stubShare object will be the child process's reference to the share server.

3 Examples & notes

        # example #1
        # hello world with remoteD
        import remoteD, time

        SharedD = remoteD.initShare()

        def child_function(Shared, arg1, arg2):
            Shared["myresult"] = "Hello World"
        
        SharedD.newProc(child_function, [arg1, arg2])

        while not SharedD.has_key("myresult"):
            time.sleep(0.2)

        print "The other process got '" + SharedD["myresult"] + "' as the answer"

Notes and considerations.

remoteD is fast - over 1.5k processes were created and connected to a single share-server on a 1GHz P3 machine using UNIXSOCK sockets, very little slowdown was observed. Slowdown occurs when physical memory is exhausted by the number of processes running.
Individual reads and writes to a remoteD share-server are atomic, so key-level locking is not explicitly provided - it exists implicitly in the way remoteD works. However, order of operations between processes is not guaranteed (just as in a multi threaded program). If you need consistent access to multiple keys, using .Lock() and .Unlock() is recommended.
Remember that objects in the share-server are remote, and attempting to create a reference to one will instead copy its value. For example: 
        SharedD['bar'] = 5
        foo = SharedD['bar']  # foo now contains the value 5
        foo += 2  # foo is now 7, SharedD['bar'] is still 5
Thus you must explicitly write changes back to the server when you are complete, in this case you'd just assign back to SharedD['bar'].
While reads and writes are fast, using them willy-nilly can cause contention at the server. Consider the following:
        foo = SharedD['bar'] / SharedD['sue']
        print str(SharedD['bar']) + " divided by " + str(SharedD['sue']) + " is " + foo
Each key is accessed twice on the server. Instead, you should do:
        bar = SharedD['bar']
        sue = SharedD['sue']
        foo = bar / sue
        print str(bar) + " divided by " + str(sue) + " is " + str(foo)
This is faster since local copies of the variables are used. Additionally, if another process were to change the value of 'sue' or 'bar' between the division and print line in the original example you could see an inconsistent answer printed, possibly "5 divided by 8 is 2"!
Remember that only individual operations are atomic, thus the following may not function as expected if .Lock() and .Unlock() are not used: 
        Shared['bar'] = Shared['bar'] + 1
Other processes could be futzing with bar between the read and the write to the share server.
Use .Lock() and .Unlock() carefully - no provision exists to auto-unlock the server is a child process halts while holding a lock. Use try clauses and always do an .Unlock() if you get an exception.