From lists at designmedia.com  Wed Jan  3 19:31:33 2007
From: lists at designmedia.com (Henry Kwan)
Date: Wed, 3 Jan 2007 16:31:33 -0800 (PST)
Subject: [Pymilter] Howto for pymilter?
In-Reply-To: <20061219170001.9836.88563.Mailman@spidey.bmsi.com>
References: <20061219170001.9836.88563.Mailman@spidey.bmsi.com>
Message-ID: <2648.192.168.1.231.1167870693.squirrel@webmail.designmedia.com>

> Oh dear.  I am so sorry, this is *really* not clear on the download page.
> The python-pydns-2.3.0 RPM is for the system python (2.3 on Centos-4).
> The
> pydns-2.3.0-2.4 RPM is for python2.4.  I apologize for the inconsistent
> naming.  (The python-pydns naming was chosen by the
> Fedora Core maintainer).

Hi,

Sorry but I was busy during the holidays and didn't get back to trying
this until now.  So I removed python-pydns-2.3.0-1 (along with
python-pyspf-2.0.1-1 since there was a dependency) and installed
pydns-2.3.0-2.4.  But when I tried to reinstall python-pyspf-2.0.1-1, I
got a failed dependency error.

"python-pydns is needed by python-pyspf-2.0.1-1.noarch"

So I installed python-pydns-2.3.0-1 again.  And I started the milter using
the shell script.  But it seems to die immediately:

[root at boxen rpm]# /etc/init.d/milter start
Starting milter: [  OK  ]
[root at boxen rpm]# /etc/init.d/milter status
milter dead but subsys locked
[root at boxen rpm]# cat /var/log/milter/milter.log
2007Jan03 16:16:35 bms milter startup

I have these packages installed on my fairly vanilla CentOS 4.4 box.

milter-0.8.6-2.RH7.i386.rpm
pysrs-0.30.11-2.noarch.rpm
python-pydns-2.3.0-1.noarch.rpm
pydns-2.3.0-2.4.noarch.rpm
python2.4-2.4.4c1-1pydotorg.i386.rpm
python-pyspf-2.0.1-1.noarch.rpm

Something seems to be borked but I'm not sure where to look.

Thanks for being so patient with me.




From stuart at bmsi.com  Wed Jan  3 20:45:15 2007
From: stuart at bmsi.com (Stuart D. Gathman)
Date: Wed, 3 Jan 2007 20:45:15 -0500 (EST)
Subject: [Pymilter] Howto for pymilter?
In-Reply-To: <2648.192.168.1.231.1167870693.squirrel@webmail.designmedia.com>
Message-ID: <Pine.LNX.4.44.0701032042380.3625-100000@bmsred.bmsi.com>

On Wed, 3 Jan 2007, Henry Kwan wrote:

> this until now.  So I removed python-pydns-2.3.0-1 (along with
> python-pyspf-2.0.1-1 since there was a dependency) and installed
> pydns-2.3.0-2.4.  But when I tried to reinstall python-pyspf-2.0.1-1, I
> got a failed dependency error.
> 
> "python-pydns is needed by python-pyspf-2.0.1-1.noarch"

Looking on my Centos-4.4 test system, it seems that I have both
python-pydns and pydns installed.  This is clearly an error in the RPM.
I got confused trying to make both versions, and didn't catch it when
test installing because I still had python-pydns installed.

Until I get it fixed, there is no reason not to have both pydns packages
installed - they go in /usr/lib/python2.3 and /usr/lib/python2.4 respectively.

-- 
	      Stuart D. Gathman <stuart at bmsi.com>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.



From stuart at bmsi.com  Wed Jan  3 22:41:40 2007
From: stuart at bmsi.com (Stuart D. Gathman)
Date: Wed, 3 Jan 2007 22:41:40 -0500 (EST)
Subject: [Pymilter] pygossip progress
Message-ID: <Pine.LNX.4.44.0701032231530.3919-100000@bmsred.bmsi.com>

Along with fixing the problems reported, and getting 0.8.7 out (just one
more fix/feature...), I have deployed pygossip on two production mail
servers (packaging still getting debugged - get CVS if interested).  It is
cutting content filtering load by about 25%.  About 35 messages/day on
mail.bmsi.com are getting rejected due to bad reputation.  Since only 100
- 200 get content filtered (most of the 11000 spams get rejected in SMTP
envelope), this is a significant load reduction.  I have the rejection
level set to reputation < -50 and confidence > 1.  Since confidence decays 
with time, people will eventually stop getting rejected, and if the next 
message isn't spam, their reputation will improve.

I have run the peer protocol also, but this needs a lot more work on
robustness - i.e. handling socket errors when peers become unreachable
without messing up sendmail with the delay and retrying an unreachable
peer periodically.

-- 
	      Stuart D. Gathman <stuart at bmsi.com>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.



From stuart at bmsi.com  Wed Jan  3 22:58:40 2007
From: stuart at bmsi.com (Stuart D. Gathman)
Date: Wed, 3 Jan 2007 22:58:40 -0500 (EST)
Subject: [Pymilter] Howto for pymilter?
In-Reply-To: <2648.192.168.1.231.1167870693.squirrel@webmail.designmedia.com>
Message-ID: <Pine.LNX.4.44.0701032253160.3948-100000@bmsred.bmsi.com>

On Wed, 3 Jan 2007, Henry Kwan wrote:

> [root at boxen rpm]# /etc/init.d/milter start
> Starting milter: [  OK  ]
> [root at boxen rpm]# /etc/init.d/milter status
> milter dead but subsys locked
> [root at boxen rpm]# cat /var/log/milter/milter.log
> 2007Jan03 16:16:35 bms milter startup
> 
> I have these packages installed on my fairly vanilla CentOS 4.4 box.
> 
> milter-0.8.6-2.RH7.i386.rpm
> pysrs-0.30.11-2.noarch.rpm
> python-pydns-2.3.0-1.noarch.rpm
> pydns-2.3.0-2.4.noarch.rpm
> python2.4-2.4.4c1-1pydotorg.i386.rpm
> python-pyspf-2.0.1-1.noarch.rpm
> 
> Something seems to be borked but I'm not sure where to look.

Hmm, looking on my production CentOS 4.4 box, I have:

milter-0.8.6-2.RH7.i386.rpm
pysrs-0.30.11-2.noarch.rpm
pydns-2.3.0-2.4.noarch.rpm
python2.4-2.4.4c1-1pydotorg.i386.rpm
pyspf-2.0.1-1.py24

Sorry, I didn't notice you *did* have both pyspf RPMs installed, and in
fact, I don't have both installed on my production system.

I also notice I didn't upload pyspf-2.0.1-1.py24 to sourceforge.  I need
to get pyspf-2.0.2 wrapped (minor bug fixes), but I'll just copy it up there
now...

-- 
	      Stuart D. Gathman <stuart at bmsi.com>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.



From stuart at bmsi.com  Thu Jan  4 15:19:31 2007
From: stuart at bmsi.com (Stuart D. Gathman)
Date: Thu, 4 Jan 2007 15:19:31 -0500 (EST)
Subject: [Pymilter] milter-0.8.7 and pyspf-2.0.2
Message-ID: <Pine.LNX.4.44.0701041515080.8222-100000@bmsred.bmsi.com>

Ok, didn't get stuff done like move bms.py to /usr/lib/pymilter (kind
of funky to have it in the log directory), but I released milter-0.8.7.

milter-0.8.7 no longer includes spf.py, you must install pyspf to get SPF
checking in the bms.py milter.

The next package split will move the milter modules to a 'pymilter' package.
This will be much better for folks who have their own milter applications
based on pymilter.

The bms.py milter application will continue to be called 'milter'.

I also released pyspf-2.0.2, which fixes some minor Received-SPF formatting
bugs and updates some docs.

-- 
	      Stuart D. Gathman <stuart at bmsi.com>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.



From stuart at bmsi.com  Fri Jan  5 22:28:34 2007
From: stuart at bmsi.com (Stuart D. Gathman)
Date: Fri, 5 Jan 2007 22:28:34 -0500 (EST)
Subject: [Pymilter] spfmilter.py
Message-ID: <Pine.LNX.4.44.0701052219440.16533-100000@bmsred.bmsi.com>

I have committed a sample spfmilter.py to pymilter CVS.  This should make 
a better starting point for those who just want to check SPF, and don't 
want all the other (effective but crufty) stuff in bms.py.  It has a basic 
policy of rejecting FAIL and PERMERROR with appropriate messages, and 
giving a 451 for TEMPERROR.

http://sourceforge.net/cvs/?group_id=139894

To be really useful, there should be a config file.  But I don't want to 
obscure the instructive value of the code.

-- 
	      Stuart D. Gathman <stuart at bmsi.com>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.




From lists at designmedia.com  Mon Jan  8 19:54:52 2007
From: lists at designmedia.com (Henry Kwan)
Date: Mon, 8 Jan 2007 16:54:52 -0800 (PST)
Subject: [Pymilter] Howto for pymilter?
In-Reply-To: <20070104170002.7161.83937.Mailman@spidey.bmsi.com>
References: <20070104170002.7161.83937.Mailman@spidey.bmsi.com>
Message-ID: <3428.192.168.1.231.1168304092.squirrel@webmail.designmedia.com>

> Message: 4
> Date: Wed, 3 Jan 2007 22:58:40 -0500 (EST)
> From: "Stuart D. Gathman" <stuart at bmsi.com>
> To: Henry Kwan <lists at designmedia.com>
> cc: pymilter at bmsi.com
> Subject: Re: [Pymilter] Howto for pymilter?
>
>
> Hmm, looking on my production CentOS 4.4 box, I have:
>
> milter-0.8.6-2.RH7.i386.rpm
> pysrs-0.30.11-2.noarch.rpm
> pydns-2.3.0-2.4.noarch.rpm
> python2.4-2.4.4c1-1pydotorg.i386.rpm
> pyspf-2.0.1-1.py24
>

Hi Stuart,

Ok, I grabbed the new packages and started the milter .  I then edited
/etc/mail/pymilter.cfg and inserted the following into my sendmail.mc and
rebuilt sendmail.cf:

INPUT_MAIL_FILTER(`milter', `S=unix:/var/run/milter/pythonsock, F=T,
T=S:240s;R:240s;E:5m')dnl

The milter seems to be running since I see this new header in my emails:

Received-SPF: none

But I have a couple of questions.

1.  The SPF DSN is sent at least once for domains that don't publish a
SPF.  How do I stop it this behavoir?

2.  Wiretap isn't working.  Or I don't understand how it's suppose to
work.  I have "mail_archive = /var/mail/mail_archive" in pymilter.cfg but
nothing ever gets dumped into /var/mail/mail_archive.

3.  The SRS part doesn't seem to work as whenever I try to start
/etc/init.d/pysrs, I get this in /var/log/milter/pysrs.log:

[root at boxen milter]# cat pysrs.log
Traceback (most recent call last):
  File "pysrs.py", line 153, in ?
    main(sys.argv[1:])
  File "pysrs.py", line 132, in main
    daemon.server.fwdomain = cp.get('srs','fwdomain',None)
  File "/usr/lib/python2.4/ConfigParser.py", line 520, in get
    raise NoOptionError(option, section)
ConfigParser.NoOptionError: No option 'fwdomain' in section: 'srs'

Thanks.





From stuart at bmsi.com  Mon Jan  8 21:03:48 2007
From: stuart at bmsi.com (Stuart D. Gathman)
Date: Mon, 8 Jan 2007 21:03:48 -0500 (EST)
Subject: [Pymilter] Howto for pymilter?
In-Reply-To: <3428.192.168.1.231.1168304092.squirrel@webmail.designmedia.com>
Message-ID: <Pine.LNX.4.44.0701082007500.10266-100000@bmsred.bmsi.com>

On Mon, 8 Jan 2007, Henry Kwan wrote:

> The milter seems to be running since I see this new header in my emails:
> 
> Received-SPF: none

Hopefully, more stuff follows that.  Otherwise, something is wrong.
It should look like this:

2007Jan08 20:58:27 [2137] Received-SPF: none (mail.bmsi.com: 209.190.247.152 is neither permitted nor denied by domain of calypso.tux.org) client_ip=209.190.247.152; envelope_from="novalug-bounces at calypso.tux.org"; helo=calypso.tux.org; receiver=mail.bmsi.com; identity=mailfrom

You now have milter-0.8.7, correct?

> But I have a couple of questions.
> 
> 1.  The SPF DSN is sent at least once for domains that don't publish a
> SPF.  How do I stop it this behavoir?

The SPF response is controlled by /etc/mail/access.  Responses are
OK, CBV, and REJECT.  CBV sends the DSN.

You can change the defaults.  For instance, I have:

SPF-None:	REJECT
SPF-Neutral:	CBV
SPF-Softfail:	CBV
SPF-Permerror:	CBV

I have best_guess = 1, so SPF none is converted to PASS/NEUTRAL for policy
lookup, and 3 strikes (no PTR, no HELO, no SPF) becomes "SPF NONE" for local
policy purposes (the Received-SPF header always shows the official SPF
result.)

You can change the default for specific domains:

# these guys aren't going to pay attention to CBVs anyway...
SPF-None:cia.gov	REJECT
SPF-None:fbi.gov	REJECT
SPF-Neutral:aol.com	REJECT
SPF-Softfail:ebay.com	REJECT

> 2.  Wiretap isn't working.  Or I don't understand how it's suppose to
> work.  I have "mail_archive = /var/mail/mail_archive" in pymilter.cfg but
> nothing ever gets dumped into /var/mail/mail_archive.

Does 'mail' have write access?  That should be logged as a traceback
in milter.log if not.

> 3.  The SRS part doesn't seem to work as whenever I try to start
> /etc/init.d/pysrs, I get this in /var/log/milter/pysrs.log:
> 
> [root at boxen milter]# cat pysrs.log
> Traceback (most recent call last):
>   File "pysrs.py", line 153, in ?
>     main(sys.argv[1:])
>   File "pysrs.py", line 132, in main
>     daemon.server.fwdomain = cp.get('srs','fwdomain',None)
>   File "/usr/lib/python2.4/ConfigParser.py", line 520, in get
>     raise NoOptionError(option, section)
> ConfigParser.NoOptionError: No option 'fwdomain' in section: 'srs'

You need to specify the forward domain - i.e. the domain you want
SRS to rewrite stuff too.

For instance, I have:

# sample SRS configuration
[srs]
secret = don't you wish
maxage = 8
hashlength = 5
;database=/var/log/milter/srs.db
fwdomain = bmsi.com
sign=bmsi.com,mail.bmsi.com,gathman.org
srs=bmsaix.bmsi.com,bmsred.bmsi.com,stl.gathman.org,bampa.gathman.org

-- 
	      Stuart D. Gathman <stuart at bmsi.com>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.



From stuart at bmsi.com  Tue Jan  9 15:33:31 2007
From: stuart at bmsi.com (Stuart D. Gathman)
Date: Tue, 9 Jan 2007 15:33:31 -0500 (EST)
Subject: [Pymilter] Howto for pymilter?
In-Reply-To: <3428.192.168.1.231.1168304092.squirrel@webmail.designmedia.com>
Message-ID: <Pine.LNX.4.44.0701091530400.15377-100000@bmsred.bmsi.com>

On Mon, 8 Jan 2007, Henry Kwan wrote:

> But I have a couple of questions.

Thank you so much for taking the time to install the packages and
report packaging bugs.  I have updated the FAQ for pymilter with
answers to your questions.  See if it helps.  Packaging and docs are
so important, but time is very limited (unless someone wants to
sponsor development at a rate my company likes).

http://bmsi.com/python/faq.html

-- 
	      Stuart D. Gathman <stuart at bmsi.com>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.



From stuart at bmsi.com  Wed Jan 10 12:41:57 2007
From: stuart at bmsi.com (Stuart D. Gathman)
Date: Wed, 10 Jan 2007 12:41:57 -0500 (EST)
Subject: [Pymilter] Howto for pymilter?
In-Reply-To: <Pine.LNX.4.44.0701082007500.10266-100000@bmsred.bmsi.com>
Message-ID: <Pine.LNX.4.44.0701101241140.20754-100000@bmsred.bmsi.com>

On Mon, 8 Jan 2007, Stuart D. Gathman wrote:

> > 1.  The SPF DSN is sent at least once for domains that don't publish a
> > SPF.  How do I stop it this behavoir?
> 
> The SPF response is controlled by /etc/mail/access.  Responses are
> OK, CBV, and REJECT.  CBV sends the DSN.

I forgot to add that the access_file option in [spf] section controls
the location of the access file.

-- 
	      Stuart D. Gathman <stuart at bmsi.com>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.



From stuart at bmsi.com  Wed Jan 17 09:10:44 2007
From: stuart at bmsi.com (Stuart D. Gathman)
Date: Wed, 17 Jan 2007 09:10:44 -0500 (EST)
Subject: [Pymilter] pygossip-0.2
Message-ID: <Pine.LNX.4.44.0701170905410.16044-100000@bmsred.bmsi.com>

I released pygossip-0.2.

I am using it in production now with a local database.  In that mode,
it makes a fine addition to a content filter.  After too many 
emails are quarantined (gossip score -50,2 or worse), we start rejecting
at MAIL FROM.  The confidence decays over time, eventually giving the 
sender another chance.

I use one of the following identity qualifiers:

SPF     for SPF PASS
GUESS   for guessed PASS (so when they get an SPF record, they start 
over).
HELO    when HELO is valid via RFC2821 (resolves to connect IP) or SPF.
x.x.x.x otherwise.

I have also been running the peer protocol between two servers.  But the
spam domain sets seem very disjoint.  There is no benefit so far as
neither has heard of the others domains.  When using the peer protocol, the
server end goes into a loop occasionally - apparently when a client
disconnects.

-- 
	      Stuart D. Gathman <stuart at bmsi.com>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.




From novel at FreeBSD.org  Thu Jan 18 02:43:21 2007
From: novel at FreeBSD.org (Roman Bogorodskiy)
Date: Thu, 18 Jan 2007 10:43:21 +0300
Subject: [Pymilter] python segfaults under heavy load
Message-ID: <20070118074321.GA88271@underworld.novel.ru>

Hi,

I'm implementing greylist milter based on py-milter and mysql as a
database backend. On my test server (low load) it works like a charm.
When I install it on a production server under heavy load, it works
several hours and then python segfaults (there's no traces/errors from
by app before it). Any suggestions what's wrong? It would be kinda
boring to rebuild python with debug symbols on a production system,
maybe that's another way to trace the problem?

Roman Bogorodskiy
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 305 bytes
Desc: not available
URL: <http://gathman.org/pipermail/pymilter/attachments/20070118/529608dc/attachment.sig>

From stuart at bmsi.com  Thu Jan 18 12:58:11 2007
From: stuart at bmsi.com (Stuart D. Gathman)
Date: Thu, 18 Jan 2007 12:58:11 -0500 (EST)
Subject: [Pymilter] python segfaults under heavy load
In-Reply-To: <20070118074321.GA88271@underworld.novel.ru>
Message-ID: <Pine.LNX.4.44.0701181244380.24186-100000@bmsred.bmsi.com>

On Thu, 18 Jan 2007, Roman Bogorodskiy wrote:

> I'm implementing greylist milter based on py-milter and mysql as a
> database backend. On my test server (low load) it works like a charm.
> When I install it on a production server under heavy load, it works
> several hours and then python segfaults (there's no traces/errors from
> by app before it). Any suggestions what's wrong? It would be kinda
> boring to rebuild python with debug symbols on a production system,
> maybe that's another way to trace the problem?

Can your production system display the total number of threads active?
Is memory growing without bound on the production system?
Is there a problem with leaking database handles?

A memory leak can be made more obvious on a test system by having milter
module allocate a big block of unused memory for each context.
We did all that many versions ago with a high volume user, and
fixed a leak.  I had 0.8.5 running full tilt (200000 connections / day
on a 450Mhz AMD) recently, so I am not aware of any breakage since then.
Let me check changes to miltermodule.c in CVS...

Other places to suspect a problem are the mysql module, libmilter, or even
(heaven forbid) python itself.  Let's start by getting versions
of mysql python module (mysql version is not as important since python
is crashing), sendmail version that supplied libmilter, and python.

-- 
	      Stuart D. Gathman <stuart at bmsi.com>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.



From stuart at bmsi.com  Wed Jan 24 13:39:45 2007
From: stuart at bmsi.com (Stuart D. Gathman)
Date: Wed, 24 Jan 2007 13:39:45 -0500 (EST)
Subject: [Pymilter] python segfaults under heavy load
In-Reply-To: <Pine.LNX.4.44.0701241324400.2997-100000@bmsred.bmsi.com>
Message-ID: <Pine.LNX.4.44.0701241337140.2997-100000@bmsred.bmsi.com>

On Wed, 24 Jan 2007, Stuart D. Gathman wrote:

> warning, no less.  Can you pull miltermodule.c 1.9 from sourceforge CVS and
> substitute it on your production system?  If pymilter is the culprit,
> this is the likely change that did it.  Or maybe you can spot a problem
> in "diff to previous version 1.9".
> 
> http://pymilter.cvs.sourceforge.net/pymilter/milter/miltermodule.c?view=log

I see my change assumes that an SMFICTX is only accessed by one thread
(hence no need for locking around use of self->t as a flag).
I suspect that that assumption may be wrong.  If reverting to 1.9 
fixes your problem, that will comfirm the suspicion.

-- 
	      Stuart D. Gathman <stuart at bmsi.com>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.