[Pymilter] Number of threads

[Stuart D. Gathman]

On Sat, 25 Dec 2004, Roman Suzi wrote:

> my milter is experiencing a number of threads problem (thread_create() fails
> after 1020 threads are created, on RedHat 7.3, Python 2.3.4). It seem to
> happen in milter C code, not in Python milter class.

What version of pymilter are you running?  Beginning with 0.6.7, runmilter()
should throw a milter.error("out of thread resources").

> What can I do to diagnose the problem before too late?
> It seems that spammers know how to DoS all kinds of milter
> by making empty connects or something like that to overflow
> the number of threads.

I had a similar problem.  It turned out that certain Python threads 
were taking a long time due to locking issues with a bayesian filter.
Sendmail already throttles the connect rate, and as long as the average
elapsed time of a milter thread stays under what sendmail expects, then
there is no problem.

I solved my locking problem by using a Python mutex to prevent multiple
milter threads from competing for the external file lock, and by skipping
bayesian content filtering for large messages.

> I'd liked to call a Python function to get the number of
> already created threads and if the number is, say > 900,
> stop accepting some connections.

import threading

if threading.activeCount() > 900:
  coverEars()

activeCount is somewhat expensive, since it counts the enumeration of
active threads.

In your Milter derived class, you could also increment a count in connect(),
and decrement in close().

I would focus on discovering what is making the milter threads take so long.

-- 
	      Stuart D. Gathman <stuart at bmsi.com>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flamis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.