[Pymilter] Forking pymilter?

[Stuart D. Gathman]

On Wed, 24 Oct 2007, Andrew Azarov wrote:

> Is there any way to make a forking python milter?

Short answer:

  Sure.  Call os.fork().

Long answer:

  The above would need a process pool to be reasonably efficient.
Libmilter, on which pymilter is based, is threaded, so the libmilter
process must have a thread per open connection.  Those threads could
be thin wrappers talking to forked servers, however.

The real question is, why?  Depending on the goal, other solutions
may be better.  Possible reasons:

1) python threading doesn't utilize multiple CPUs.
   At 200000 connections per day, pymilter CPU for my milter, including
   bayesian content filtering, is neglible on my low end Dell SC440.
   Even if it weren't, there are plenty of other processes to keep
   dual cores busy.  If you're looking to process multi-millions of messages
   per day, you could use multiple low end machines instead of one
   super computer.  If you do want the single super computer running
   pymilter, feel free to contribute the process pool manager. :-)

2) You are worried about memory corruption.  
   Pure python is fairly immune to that. 

3) You are worried about too many threads.
   There is one thread per open connection (unless you create more).  Sendmail
   limits open connections.  There was a bug where some libmilter threads
   weren't getting recycled, but that seems to be fixed now.  A forking server
   would still be subject to that bug.

-- 
	      Stuart D. Gathman <stuart at bmsi.com>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.