[Pymilter] Number of threads

[Roman Suzi]

On Mon, 27 Dec 2004, Stuart D. Gathman wrote:

Thank you for your reply, Stuart!

> On Sat, 25 Dec 2004, Roman Suzi wrote:
>
> > my milter is experiencing a number of threads problem (thread_create() fails
> > after 1020 threads are created, on RedHat 7.3, Python 2.3.4). It seem to
> > happen in milter C code, not in Python milter class.
>
> What version of pymilter are you running?  Beginning with 0.6.7, runmilter()
> should throw a milter.error("out of thread resources").

I am using 0.7.0, highly customized.

> > What can I do to diagnose the problem before too late?
> > It seems that spammers know how to DoS all kinds of milter
> > by making empty connects or something like that to overflow
> > the number of threads.
>
> I had a similar problem.  It turned out that certain Python threads
> were taking a long time due to locking issues with a bayesian filter.

I do not use bayesian filter.

> Sendmail already throttles the connect rate, and as long as the average
> elapsed time of a milter thread stays under what sendmail expects, then
> there is no problem.
>
> I solved my locking problem by using a Python mutex to prevent multiple
> milter threads from competing for the external file lock, and by skipping
> bayesian content filtering for large messages.
>
> > I'd liked to call a Python function to get the number of
> > already created threads and if the number is, say > 900,
> > stop accepting some connections.
>
> import threading
>
> if threading.activeCount() > 900:
>   coverEars()
>
> activeCount is somewhat expensive, since it counts the enumeration of
> active threads.
>
> In your Milter derived class, you could also increment a count in connect(),
> and decrement in close().

Good idea, thanks.

> I would focus on discovering what is making the milter threads take so long.

Mail comes in bursts. Sometimes milter is dead in 3 minutes, because
30 * 60 * 3 = 5400 - which is > 1020 five times.
(connection throttle in sendmail = 30)

Sincerely yours, Roman A.Suzi
-- 
 - Petrozavodsk - Karelia - Russia - mailto:rnd at onego.ru -


<!DSPAM:10114CDCA6E75016461648967>