[Pymilter] Forking pymilter?

[Dwayne Litzenberger]

On October 24, 2007 01:54:09 pm Stuart D. Gathman wrote:
> Is there any concurrency difference between a single thread calling
> python-ldap on behalf of multiple threads, and multiple threads calling
> python-ldap with a mutex?  If you want an ansynchronous call (start query,
> wait for query to finish and get result), then simply have the wrapper
> return a lazy-result proxy object, which waits for the result when an
> attribute is queried.

My LDAP thread is a bit of a hack that emulates synchronous behaviour using 
the asynchronous calls provided by python-ldap[1].  The thread starts by 
waiting for a serialized request on a Queue.Queue (using Queue.get() with an 
infinite timeout).  Once the LDAP thread receives a request, it makes the 
request via the asynchronous LDAPObject.search() function, and stores the 
resulting msgid in a 'pendingRequests' dictionary (along with the request 
itself, and a per-request Queue object used to deliver the result).  It then 
begins alternately polling the request queue and LDAPObject.result2(), with 
10ms timeouts for each.  (The polling is done because we don't have something 
like WaitForMultipleObjects that would allow us to wait on both the 
Queue.get() and the LDAPObject.result2() calls.)

When the thread gets a response from LDAPObject.result2() (which might be an 
exception), it looks up the msgid in the pendingRequests dictionary, 
serializes the response, and sends it over the per-request Queue to the 
calling thread, which decodes it and returns the result (or raises the 
associated exception).

There was also some logic to attempt to reconnect to the server and re-issue 
the pending searches if the connection died.  That was also more complicated 
than it would have been had I been able to make real synchronous calls.

> Are you forking several processes to serve ldap query 
> requests?  That would certainly be a win, performance wise - and I can see
> it would require some effort to wrap the calls (is there a python RMI
> equivalent?)

I didn't actually do any forking.  os.fork() duplicates all running threads, 
right?

I've never heard of any Python RMI equivalent, which is why I more-or-less 
ended up inventing one for this project.

Cheers,
 - Dwayne


[1] See http://python-ldap.sourceforge.net/doc/python-ldap/ldap-objects.html
-- 
Dwayne Litzenberger, B.A.Sc.
Information Technology Analyst

Open Systems Canada Limited
1627 Broad Street
Regina, SK S4P1X3
Office: 306.359.6725
http://www.oscl.ca/