[Pymilter] Forking pymilter?

[Stuart D. Gathman]

On Wed, 24 Oct 2007, Dwayne Litzenberger wrote:

> python-ldap already wraps its API in a mutex.  That's useless, though, because 
> when you make a call to, say, ldap.search_s ("_s" stands for synchronous), 
> python-ldap acquires a library-wide lock.  Any other thread that tries to 
> access the ldap library in the meantime (including to access other LDAP 
> servers) blocks until the previous call completes.  At that point, we've done 
> away with concurrency, which defeats the purpose of using threads in the 
> first place.

This is another example of globals == evil.  But it sounds like a process-pool
class would be a useful addition to pymilter.

> I ended up spending a lot of time writing code to pass messages between the 
> threads created by libmilter and a thread I created specifically to access 
> python-ldap.  If I had been able to fork a server for each connection, I 
> could have simply used ldap.search_s and been done with it.

Is there any concurrency difference between a single thread calling python-ldap
on behalf of multiple threads, and multiple threads calling python-ldap
with a mutex?  If you want an ansynchronous call (start query, wait for query
to finish and get result), then simply have the wrapper return a lazy-result 
proxy object, which waits for the result when an attribute is queried.
Are you forking several processes to serve ldap query requests?  That would
certainly be a win, performance wise - and I can see it would require
some effort to wrap the calls (is there a python RMI equivalent?)

> libmilter demands a particular threading model that happens to be
> incompatible with OpenLDAP.  What's not clear to me is why the Sendmail
> people thought this was necessary.  IMHO, it would have been better to
> publish a document specifying how to communicate over the milter socket, and
> to provide automatic thread creation as an optional library feature.

Yeah.  Especially since libmilter theading is incompatible with
Java VM threading.  That is what drove me to python milter - python
could handle libmilter threading, perl at the time could not, ruby
at the time could not, and Java threads are not compatible (signal use, etc).
And I wanted a high level garbage collected language.  It turned out
to be a good choice.  Python is higher level than Java, CPU has not been an
issue, at least with my milters, and it is easy to add C libraries to python.

If there is a threadsafe C ldap API, it would not be hard
to wrap it for a threadsafe python API.

-- 
	      Stuart D. Gathman <stuart at bmsi.com>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.