From stuart at bmsi.com  Tue Nov  9 17:57:47 2004
From: stuart at bmsi.com (Stuart D. Gathman)
Date: Tue, 9 Nov 2004 17:57:47 -0500 (EST)
Subject: [Pymilter] Malformed attachment problem
Message-ID: <Pine.LNX.4.44.0411091738140.28579-100000@bmsred.bmsi.com>

I have been trying to improve the handling of malformed attachments by the
mime.py module.  Currently, a malformed attachment (missing boundaries, etc)
gets an exception, and is rejected as spam.  However, I keep getting cases
where someone forwards a message (with missing boundaries) as an rfc822
attachment.  I don't know whether the sender received the message garbled, and
read it with a tolerant email client, or their email client garbled it
while forwarding.  

This usually happens internally (where everyone uses OutHouse), so I added a
feature to pymilter so that a malformed attachment exception does not block
internal mail.

Here is my attempt to postphone parsing of rfc822 attachments until needed:

***************
*** 213,225 ****
              except Errors.HeaderParseError:
                  msg = self._class()
                  self._parsebody(msg, fp)
!           except Errors.BoundaryError:
!               part = container.get_payload(decode=True)
!               bl = re.compile(r'(?:\r\n|\r|\n){2}')
!               parthdrs, part = bl.split(part, 1)
!               msg = self.parsestr(parthdrs, headersonly=1)
!               msg.set_payload(part)
!           container.set_payload([msg])
          else:
              text = fp.read()
              if firstbodyline is not None:
--- 206,212 ----
              except Errors.HeaderParseError:
                  msg = self._class()
                  self._parsebody(msg, fp)
!             container.set_payload([msg])
          else:
              text = fp.read()
              if firstbodyline is not None:


I wanted malformed rfc822 attachments to produce an exception only
when the scan_rfc822 option is turned on.  Unfortunately, my test cases
indicate that with this change, malformed attachments are never
detected - even with scan_rfc822 on.

Does anyone have pointers to handling this problem with the email
package?  Suggestions?  I can make a case for always rejecting malformed
mail - but the whole point of the rfc822 attachment is to be able
to encapsulate mail, malformed or not.

-- 
	      Stuart D. Gathman <stuart at bmsi.com>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flamis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.



From stuart at bmsi.com  Tue Nov  9 18:04:22 2004
From: stuart at bmsi.com (Stuart D. Gathman)
Date: Tue, 9 Nov 2004 18:04:22 -0500 (EST)
Subject: [Pymilter] Malformed attachment problem
In-Reply-To: <Pine.LNX.4.44.0411091738140.28579-100000@bmsred.bmsi.com>
Message-ID: <Pine.LNX.4.44.0411091802190.28742-100000@bmsred.bmsi.com>

On Tue, 9 Nov 2004, Stuart D. Gathman wrote:

> Here is my attempt to postphone parsing of rfc822 attachments until needed:

Here is the additional test case in testmime.py (sorry, reversed diff):

6d6
< from email import Errors
34,44d33
<     msg = mime.MimeMessage(open('test/missingboundary',"r"))
<     # should get no exception as long as we don't try to parse
<     # message attachments
<     mime.defang(msg,scan_rfc822=False)
<     msg.dump(open('test/missingboundary.out','w'))
<     msg = mime.MimeMessage(open('test/missingboundary',"r"))
<     try:
<       mime.defang(msg)
<       self.fail('should get boundary error parsing bad rfc822 attachment')
<     except Errors.BoundaryError:
<       pass

-- 
	      Stuart D. Gathman <stuart at bmsi.com>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flamis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.



From stuart at bmsi.com  Sat Nov 20 11:51:58 2004
From: stuart at bmsi.com (Stuart D. Gathman)
Date: Sat, 20 Nov 2004 11:51:58 -0500 (EST)
Subject: [Pymilter] SPF softfail
Message-ID: <Pine.LNX.4.44.0411201144290.8094-100000@bmsred.bmsi.com>

I am about to release 0.7.2.  There are no low level changes.
The bms milter blocks SPF softfail, except for domains listed in the config.
This cans lots of spam.

It is currently blocked via TEMPFAIL (SMTP 450).  The theory is that
softfail means the domain is testing before implementing -all, so
the admin should see the error and fix it - and the mail will then go through.
It might be better to REJECT - so the end user will see the error
sooner, because not all mail admins have a clue.

A few domains are using softfail because they are too timid to
make their user authenticate, and should be using neutral instead.
You can configure the milter to accept softfails for those domains.

-- 
	      Stuart D. Gathman <stuart at bmsi.com>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flamis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.



From stuart at bmsi.com  Wed Nov 24 09:31:59 2004
From: stuart at bmsi.com (Stuart D. Gathman)
Date: Wed, 24 Nov 2004 09:31:59 -0500 (EST)
Subject: [Pymilter] Recognizing dynamic IPs
Message-ID: <Pine.LNX.4.44.0411240909370.2155-100000@bmsred.bmsi.com>

I have 0.7.2 in production on 2 machines.  Seems pretty stable.  A very 
useful feature of 0.7.2 is recognizing dynamic PTR records.  You know, 
things like "adsl-57.76.154.info.com.ph".

I have a very heuristic function to do this.  Ideas on improving it
are welcome.

# examples we don't yet recognize:
#
# 1Cust65.tnt4.atl4.da.uu.net at ('67.192.40.65', 4588)
# 1Cust141.tnt30.rtm1.nld.da.uu.net at ('213.116.154.141', 2036)
# user64.net2045.mo.sprint-hsd.net at ('67.77.185.64', 3901)
# wiley-268-8196.roadrunner.nf.net at ('205.251.174.46', 4810)
# 221.fib163.satnet.net at ('200.69.163.221', 3301)
# BHE214126.res-com.wayinternet.com.br at ('200.159.214.126', 3528)
# ZR093223.ppp.dion.ne.jp at ('222.14.93.223', 3593)
# cpc2-ches1-4-0-cust8.lutn.cable.ntl.com at ('80.4.105.8', 61099)
# docsis8-179.sby.link.net.id at ('202.137.8.179', 1440)
# user239.res.openband.net at ('65.246.82.239', 1392)
# S010600e0299501e5.rd.shawcable.net at ('24.64.82.212', 1838)
# xdsl-2449.zgora.dialog.net.pl at ('81.168.237.145', 1238)

import re

ip3 = re.compile('([0-9]{1,3})[.-]([0-9]{1,3})[.-]([0-9]{1,3})')
rehmac = re.compile('h[0-9a-f]{12}[.]|pcp[0-9]{6,10}pcs[.]|no-reverse')

def is_dynip(host,addr):
  """Return True if hostname is for a dynamic ip.
  Examples:

  >>> is_dynip('post3.fabulousdealz.com','69.60.99.112')
  False
  >>> is_dynip('adsl-69-208-201-177.dsl.emhril.ameritech.net','69.208.201.177')
  True
  """
  if addr:
    if host.find(addr) >= 0: return True
    a = addr.split('.')
    m = ip3.search(host)
    if m:
      g = list(m.groups())
      if g == a[1:] or g == a[:3]: return True
      g.reverse()
      if g == a[1:] or g == a[:3]: return True
    if rehmac.search(host): return True
    if host.find("-%s." % '-'.join(a[2:])) >= 0: return True
    if host.find("w%s." % '-'.join(a[:2])) >= 0: return True
    if host.find(''.join(a[:3])) >= 0: return True
    if host.find(''.join(a[1:])) >= 0: return True
    x = "%02x%02x%02x%02x" % tuple(map(int,a))
    if host.lower().find(x) >= 0: return True
    z = [n.zfill(3) for n in a]
    if host.find('-'.join(z)) >= 0: return True
    if host.find("-%s." % '-'.join(z[2:])) >= 0: return True
    if host.find("%s." % ''.join(z[2:])) >= 0: return True
    if host.find(''.join(z)) >= 0: return True
  return False

-- 
	      Stuart D. Gathman <stuart at bmsi.com>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flamis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.



From stuart at bmsi.com  Thu Nov 25 22:13:06 2004
From: stuart at bmsi.com (Stuart D. Gathman)
Date: Thu, 25 Nov 2004 22:13:06 -0500 (EST)
Subject: [Pymilter] Re: Python milter extension, OpenBSD Installation Notes
In-Reply-To: <4F843A9E-3EEF-11D9-B851-000D933B29EE@msys.ch>
Message-ID: <Pine.LNX.4.44.0411252202330.10364-100000@bmsred.bmsi.com>

On Thu, 25 Nov 2004, Marc Balmer wrote:

> In the README file I found an OpenBSD specific note which is 
> unfortunately wrong and thus misleading.  It is written there that 
> "Sendmail is broken on OpenBSD for unix domain sockets."  That's 
> incorrect; there is no problem at all with sendmail milters using unix 
> domain sockets on OpenBSD.

I took that statement from someone using it on openbsd.  I suppose I should have said,
"So and So says that sendmail is broken".  Since it either never was really broken or is now
fixed, I'll update it.  Thanks for the feedback.

Sorry I still haven't split out the milter application from the extension.

The milter application has a function which should be in the library section.  I am not sure
where.  It is the is_dynip(host,ip) heristic which tries to recognize PTR names which are for
dynamic IPs.  I want to put it in the spf module, but it is not really part of spf - just
a heuristic that is handy when doing spf type things.

I don't want too many top level modules.  Maybe the thing to do is to create a 
milter subpackage.  But spf is useful for other MTAs.  Packaging is hard.

-- 
	      Stuart D. Gathman <stuart at bmsi.com>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flamis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.