From stuart at gathman.org  Tue Dec 13 14:21:29 2016
From: stuart at gathman.org (Stuart D. Gathman)
Date: Tue, 13 Dec 2016 14:21:29 -0500 (EST)
Subject: [Pymilter] PyMilter question
In-Reply-To: <d4f57aa9-7171-454c-a8b2-01b22d4c89a6@laposte.net>
References: <d4f57aa9-7171-454c-a8b2-01b22d4c89a6@laposte.net>
Message-ID: <alpine.LRH.2.20.1612131349450.2542@fairfax.gathman.org>

On Tue, 13 Dec 2016, Philippe Lagadec wrote:

> Do you know if I can do it in the Milter.envrcpt method? And if so, which 
> result shall it return instead of Milter.CONTINUE, is it Milter.ACCEPT?

Milter.ACCEPT should skip all remaining callbacks.  Note that there
can be multiple recipients.  You probably want to wait until the 
DATA callback and ACCEPT unless your test recipients are the
only recipients.

> I looked for a documentation of the milter protocol/process, but I haven't 
> found any so far.

The documentation is linked from README, but is at
http://pythonhosted.org/pymilter/

> Also, I have found several code repositories for pymilter:
> https: //github.com/jmehnle/pymilter
> https: //sourceforge.net/projects/pymilter/
> https: //github.com/sdgathman/pymilter

The project is recently moved to github and converted from CVS to git.
Sourceforge has the last official release, before python3 support.
You'll also find releases at https://pypi.python.org/pypi/pymilter/

Please use the github master if you can and report any problems.  The
next release will be 1.1 and support python3.  Milters are working with
py3, but there are still choices to be made as to what is bytes/str in
the API.  I'll make a minor release (1.0.2) if you want to be sure not to 
conflict.  The rpms build for fedora and epel.

-- 
 	      Stuart D. Gathman <stuart at gathman.org>
"Confutatis maledictis, flamis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.


From tonynelson at georgeanelson.com  Wed Dec 21 00:18:31 2016
From: tonynelson at georgeanelson.com (Tony Nelson)
Date: Wed, 21 Dec 2016 00:18:31 -0500
Subject: [Pymilter] My own version of the bms.py milter
Message-ID: <1482297511.3466.0@master.lan>

I've been using (and modifying) the bms.py milter for almost 10 years
now, and it seems time to give back.  I have tarballs on my server,
that build properly with rpmbuild -ta:

     http://www.georgeanelson.com/milter-0.9.tar.gz
     http://www.georgeanelson.com/pymilter-1.0.tar.gz

They were made from the tarballs on Sourceforge (Revision 1.207
2016/04/22 16:05:14  customdesigned).  I can put my changes into git
(with some effort and study), if desired.  Probably the best way to
see what I've done is with Meld on the directories (well, I use diff
and vimdiff).

My use of the milter emphasizes detecting dynamic hosts and forbidding
them to relay (dynip() has seen much work) or lie, with my own simple
greylisting for non-relaying dynamic hosts that have real HELO names,
followed by the Spamhaus Zen and DBL rbls to filter what is left.
I've made many other changes, as well.

What I did works pretty well and much of the spam that gets through
comes from real mailservers at spam companies, so I have not used
SPF/SRS/SES or Dspam or DKIM or any of the rest.  I don't think I've
broken any of them, and I may use them someday.

BTW: according to the Milter docs, each callback during a connection
can be on a different thread, so threading().local is probably wrong.
I've fixed that.

-- 
____________________________________________________________________
TonyN.:'                       <mailto:tonynelson at georgeanelson.com>
       '                              <http://www.georgeanelson.com/>


From stuart at gathman.org  Wed Dec 21 21:34:12 2016
From: stuart at gathman.org (Stuart D. Gathman)
Date: Wed, 21 Dec 2016 21:34:12 -0500 (EST)
Subject: [Pymilter] My own version of the bms.py milter
In-Reply-To: <1482297511.3466.0@master.lan>
References: <1482297511.3466.0@master.lan>
Message-ID: <alpine.LRH.2.20.1612212119420.17834@fairfax.gathman.org>

On Wed, 21 Dec 2016, Tony Nelson wrote:

> I've been using (and modifying) the bms.py milter for almost 10 years
> now, and it seems time to give back.  I have tarballs on my server,
> that build properly with rpmbuild -ta:
>
>   http: //www.georgeanelson.com/milter-0.9.tar.gz
>   http: //www.georgeanelson.com/pymilter-1.0.tar.gz
>
> They were made from the tarballs on Sourceforge (Revision 1.207
> 2016/04/22 16:05:14  customdesigned).  I can put my changes into git
> (with some effort and study), if desired.  Probably the best way to
> see what I've done is with Meld on the directories (well, I use diff
> and vimdiff).

Development has moved to https://github.com/sdgathman/milter
(and pymilter).  The big push is working with python3.  Which it
does - just need to nail down more of the bytes/str decisions in
the libmilter interfaces.

What we really need is a framework that plugs in modules.  Or just
keep making separate milters with common libs like we have now.
It sounds like the improved "isdynip" is a win.

> What I did works pretty well and much of the spam that gets through
> comes from real mailservers at spam companies, so I have not used
> SPF/SRS/SES or Dspam or DKIM or any of the rest.  I don't think I've
> broken any of them, and I may use them someday.

SRS blocks "bounce spam".  SPF and DKIM let you ban authenticated
domains.  All the (competent) spammers have fully authenticated domains -
that they use for a few days and abandon.  I've been focusing on
sharing banned domains with a blacklist.

Furthermore, banning is relative to authentication.  If the connection
gets a SOFTFAIL, then the domain is banned for SOFTFAIL, but not for
PASS.  If the connection got a PASS - then the domain is banned
up front for everything without even bothering to check SPF.

> BTW: according to the Milter docs, each callback during a connection
> can be on a different thread, so threading().local is probably wrong.
> I've fixed that.

The threading().local is only used for a cache to avoid creating
new Greylist objects all the time - and the Greylist objects are
not threadsafe.  In general, it is for global objects that are not
threadsafe - not for connection specific objects.  It is a *good* thing
if the callbacks are on different threads.  :-)

However, that should be explained in the comments.  Obviously, anything
connection specific goes in the FooMilter object.

-- 
 	      Stuart D. Gathman <stuart at gathman.org>
"Confutatis maledictis, flamis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.


From stuart at gathman.org  Thu Dec 22 12:24:33 2016
From: stuart at gathman.org (Stuart D. Gathman)
Date: Thu, 22 Dec 2016 12:24:33 -0500 (EST)
Subject: [Pymilter] My own version of the bms.py milter
In-Reply-To: <alpine.LRH.2.20.1612212119420.17834@fairfax.gathman.org>
References: <1482297511.3466.0@master.lan>
	<alpine.LRH.2.20.1612212119420.17834@fairfax.gathman.org>
Message-ID: <alpine.LRH.2.20.1612221222460.28861@fairfax.gathman.org>

On Wed, 21 Dec 2016, Stuart D. Gathman wrote:

... some stuff about Tony's work

Your milter is not liking my direct email:

... while talking to georgeanelson.com.:
>>> MAIL From:<SRS0=eI5Tv=YF==stuart at gathman.org> SIZE=4017
<<< 451 4.3.0 pymilter: untrapped exception in pythonfilter
<tonynelson at georgeanelson.com>... Deferred: 451 4.3.0 pymilter:
untrapped exception in pythonfilter
Warning: message still undelivered after 4 hours
Will keep trying until message is 5 days old

-- 
 	      Stuart D. Gathman <stuart at gathman.org>
"Confutatis maledictis, flamis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.


From stuart at gathman.org  Thu Dec 22 12:42:09 2016
From: stuart at gathman.org (Stuart D. Gathman)
Date: Thu, 22 Dec 2016 12:42:09 -0500 (EST)
Subject: [Pymilter] My own version of the bms.py milter
In-Reply-To: <alpine.LRH.2.20.1612221222460.28861@fairfax.gathman.org>
References: <1482297511.3466.0@master.lan>
	<alpine.LRH.2.20.1612212119420.17834@fairfax.gathman.org>
	<alpine.LRH.2.20.1612221222460.28861@fairfax.gathman.org>
Message-ID: <alpine.LRH.2.20.1612221238180.3532@fairfax.gathman.org>

On Thu, 22 Dec 2016, Stuart D. Gathman wrote:

> On Wed, 21 Dec 2016, Stuart D. Gathman wrote:
>
> ... some stuff about Tony's work
>

Also, an IP is not "dynamic" for email purposes if it has a valid
HELO (i.e. looking up the alleged hostname matches the connect IP).
Many ISPs will not (or the process is inordinately expensive timewise)
assign rDNS for small static IP allocations, even for business accounts.
The whole point of HELO is to provide a verifiable hostname for exactly
this situation.

-- 
 	      Stuart D. Gathman <stuart at gathman.org>
"Confutatis maledictis, flamis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.


From tonynelson at georgeanelson.com  Thu Dec 22 13:19:31 2016
From: tonynelson at georgeanelson.com (Tony Nelson)
Date: Thu, 22 Dec 2016 13:19:31 -0500
Subject: [Pymilter] My own version of the bms.py milter
In-Reply-To: <alpine.LRH.2.20.1612212119420.17834@fairfax.gathman.org> (from
	stuart@gathman.org on Wed Dec 21 21:34:12 2016)
References: <1482297511.3466.0@master.lan>
	<alpine.LRH.2.20.1612212119420.17834@fairfax.gathman.org>
Message-ID: <1482430771.10294.1@master.lan>

On 16-12-21 21:34:12, Stuart D. Gathman wrote:
  ...
> Development has moved to https://github.com/sdgathman/milter
> (and pymilter).

Yes, I saw.

> The big push is working with python3.  Which it
> does - just need to nail down more of the bytes/str decisions in
> the libmilter interfaces.

OK.  I have no current plans to use Python3, but I know I will have
to some day.  Soon I will have to migrate two servers from CentOS5.


> What we really need is a framework that plugs in modules.  Or just
> keep making separate milters with common libs like we have now.

One of the servers I have the milter on has some special requirements,
so I subclassed the milter to provide for them.  I had to add a
self.cur_rcpt to expose canon_to to the subclass.  Which is to say, for
at least some applications, a plug-in architecture will be difficult.

> It sounds like the improved "isdynip" is a win.

Yes, I think dynip() can just be used.  As it has lots more patterns,
it returns a number that can be used to assign blame, and can be run
as a script to display the pattern.  I have found it to be reliable.


>> What I did works pretty well and much of the spam that gets through
>> comes from real mailservers at spam companies, so I have not used
>> SPF/SRS/SES or Dspam or DKIM or any of the rest.  I don't think I've
>> broken any of them, and I may use them someday.
> 
> SRS blocks "bounce spam".  SPF and DKIM let you ban authenticated
> domains.  All the (competent) spammers have fully authenticated  
> domains -
> that they use for a few days and abandon.  I've been focusing on
> sharing banned domains with a blacklist.

The (little) servers I maintain don't get (well, accept) bounce spam
(which was your work, I think), and the Spamhaus Zen and DBL work pretty
well (for spam directed at the US).  The other rbls I've tried,  
including
a couple from companies that are trying some sharing among their  
customer
base, were not helpful.  This is hard to do and takes lots of mail
servers.


> Furthermore, banning is relative to authentication.  If the connection
> gets a SOFTFAIL, then the domain is banned for SOFTFAIL, but not for
> PASS.  If the connection got a PASS - then the domain is banned
> up front for everything without even bothering to check SPF.

The spam that gets through mostly says it is from the domain it seems to
be from.  The only thing that SPF could do is trip them up if they made
a mistake -- but lots of TBTF email providers aren't to be trusted here
either (can't get rDNS right, can't get HELO right, and so on).


>> BTW: according to the Milter docs, each callback during a connection
>> can be on a different thread, so threading().local is probably wrong.
>> I've fixed that.
> 
> The threading().local is only used for a cache to avoid creating
> new Greylist objects all the time - and the Greylist objects are
> not threadsafe.  In general, it is for global objects that are not
> threadsafe - not for connection specific objects.  It is a *good*  
> thing
> if the callbacks are on different threads.  :-)

I get it.  I'll revert my "fix".

I take it that Greylist objects do not contain any connection-
specific data.

> However, that should be explained in the comments.  Obviously,  
> anything
> connection specific goes in the FooMilter object.

OK.

-- 
____________________________________________________________________
TonyN.:'                       <mailto:tonynelson at georgeanelson.com>
       '                              <http://www.georgeanelson.com/>


From tonynelson at georgeanelson.com  Thu Dec 22 12:53:06 2016
From: tonynelson at georgeanelson.com (Tony Nelson)
Date: Thu, 22 Dec 2016 12:53:06 -0500
Subject: [Pymilter] My own version of the bms.py milter
In-Reply-To: <alpine.LRH.2.20.1612221222460.28861@fairfax.gathman.org> (from
	stuart@gathman.org on Thu Dec 22 12:24:33 2016)
References: <1482297511.3466.0@master.lan>
	<alpine.LRH.2.20.1612212119420.17834@fairfax.gathman.org>
	<alpine.LRH.2.20.1612221222460.28861@fairfax.gathman.org>
Message-ID: <1482429186.10294.0@master.lan>

On 16-12-22 12:24:33, Stuart D. Gathman wrote:
> On Wed, 21 Dec 2016, Stuart D. Gathman wrote:
> 
> ... some stuff about Tony's work
> 
> Your milter is not liking my direct email:

I seem to have been a bit sleepy last night, making changes to cope
with SRS and mailing lists re. auto-whitelist.  Mail is flowing
again, and soon I'll see if my changes work. ;)

-- 
____________________________________________________________________
TonyN.:'                       <mailto:tonynelson at georgeanelson.com>
       '                              <http://www.georgeanelson.com/>



From tonynelson at georgeanelson.com  Thu Dec 22 14:39:10 2016
From: tonynelson at georgeanelson.com (Tony Nelson)
Date: Thu, 22 Dec 2016 14:39:10 -0500
Subject: [Pymilter] My own version of the bms.py milter
In-Reply-To: <alpine.LRH.2.20.1612221238180.3532@fairfax.gathman.org> (from
	stuart@gathman.org on Thu Dec 22 12:42:09 2016)
Message-ID: <1482435550.10520.0@master.lan>

On 16-12-22 12:42:09, Stuart D. Gathman wrote:
> On Thu, 22 Dec 2016, Stuart D. Gathman wrote:
> 
>> On Wed, 21 Dec 2016, Stuart D. Gathman wrote:
>> 
>> ... some stuff about Tony's work
>> 
> 
> Also, an IP is not "dynamic" for email purposes if it has a valid
> HELO (i.e. looking up the alleged hostname matches the connect IP).
> Many ISPs will not (or the process is inordinately expensive timewise)
> assign rDNS for small static IP allocations, even for business  
> accounts.

The ISPs I know have automated the process.  It doesn't seem to be
business class if the ISP doesn't provide for proper site configuration.
I understand that it won't be provided for home users.

I never accept mail from connections with no rDNS.  That can happen
for transient failures, so I TEMPFAIL it for a while and then REJECT
a message (if they can actually send one) so a legitimate user can
find out.

> The whole point of HELO is to provide a verifiable hostname for  
> exactly
> this situation.

Many spammers now have domain names for their bots and stolen VPSs.

If the rDNS seems dynamic but the HELO verifies, I greylist the
message (by message-id, so TBTF can also send email).  Some spammers
actually handle that (but many do it badly, trying 3 times in close
succession -- even in the same connection).  Real mailservers use
exponential back-off, and slide into the "acceptable" time window.
At least in my limited experience over the past 9 years, this has
been the case and does not reject real mail.

My greylisting had been delaying mail from your site, as the auto-
whitelist didn't handle changing the sender ("-bounces") or removing
SRS without using SRS.  My attempts to handle that while tired
resulted in those tracebacks.  It may be working now.

At least I assume SRS tries to remove SRS it didn't generate, but I
didn't follow the code all the way.  The "spec" for SRS at
<http://www.libsrs2.org/srs/srs.pdf> notes that SRS0 is "opaque",
meaning that there were multiple forms in the wild, and I can only
remove the "standard" one.  Yours is the only site I've gotten SRS
from, so I didn't have experience with it.

-- 
____________________________________________________________________
TonyN.:'                       <mailto:tonynelson at georgeanelson.com>
       '                              <http://www.georgeanelson.com/>


From stuart at gathman.org  Fri Dec 23 11:33:04 2016
From: stuart at gathman.org (Stuart D. Gathman)
Date: Fri, 23 Dec 2016 11:33:04 -0500 (EST)
Subject: [Pymilter] My own version of the bms.py milter
In-Reply-To: <1482435550.10520.0@master.lan>
References: <1482435550.10520.0@master.lan>
Message-ID: <alpine.LRH.2.20.1612231121200.9046@fairfax.gathman.org>

(NOTE, I really appreciate the use of and work on the milter code.  The
spirited debate on the best anti-spam heuristics should be taken in the
proper spirit!  )

On Thu, 22 Dec 2016, Tony Nelson wrote:

>> Also, an IP is not "dynamic" for email purposes if it has a valid
>> HELO (i.e. looking up the alleged hostname matches the connect IP).
>> Many ISPs will not (or the process is inordinately expensive timewise)
>> assign rDNS for small static IP allocations, even for business accounts.
>
> The ISPs I know have automated the process.  It doesn't seem to be
> business class if the ISP doesn't provide for proper site configuration.
> I understand that it won't be provided for home users.
>
> I never accept mail from connections with no rDNS.  That can happen
> for transient failures, so I TEMPFAIL it for a while and then REJECT
> a message (if they can actually send one) so a legitimate user can
> find out.

The irony of that heuristic is that my real business 2 yr contract 
for $100/mo has no rDNS, while spammers can rent a VPS for < $10 a month
(they'll abandon it within a few days) with rDNS.

Yes, I've considered forwarding outgoing mail through a VPS just to get
the rDNS.  But having done that for clients, I find that spammers
have just got done renting that IP, and it is on multiple blacklists. 
IP blacklists are *so* obsolete.  Now that spammers have been forced
to use authentic (if ephemeral) domains, domain blacklists are what
you want.  IPs are only used briefly by spammers as well, and
blacklisting an IP just injures the innocent party that inherits it.

Plus I'm philosophically opposed to the common policy of requiring
rDNS for email and ignoring HELO.  If people don't want my personal
email, so be it.

-- 
 	      Stuart D. Gathman <stuart at gathman.org>
"Confutatis maledictis, flamis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.