From stuart at bmsi.com Wed Jun 1 11:43:28 2005 From: stuart at bmsi.com (Stuart D. Gathman) Date: Wed, 1 Jun 2005 11:43:28 -0400 (EDT) Subject: [Pymilter] RE: You are correct, it is a bug (rejecting non-SRS) In-Reply-To: <20050526065736.37BCED0025C@2.revivim.org.il> Message-ID: On Thu, 26 May 2005, Alex Savguira wrote: > Thanks. I am glad we have come to the conclusion on it... I would rather have > your term, I am on kinda unfriendly terms with Python, never actually studied > it, or wrote a line of code in it, I can read it a bit, out of common sense, > but still.. Would save me tons of struggling with improper indentation (this > I figured out already ;0) etc... Thanks The latest CVS is now on sourceforge. It is not released yet to make sure all the loose ends connect, but that version of bms.py is running on my system. > P.S. I like your idea of doing the extra sanity checks on attachments, would > you consider adding the simple content check for attached archives? Simple > diverting of zip files containing the unsafe extensions type will be > enough... Far too many viruses today send themselves zip-archived. No fancy Yes, that is straightforward in python, it already supports zip archives. Since supporting the new email package in python2.4, there are bugs in the attachment scanning. Supporting earlier python versions seems to require an alternate mime.py implementation. -- Stuart D. Gathman Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154 "Confutatis maledictis, flamis acribus addictis" - background song for a Microsoft sponsored "Where do you want to go from here?" commercial. From stuart at bmsi.com Mon Jun 6 10:29:59 2005 From: stuart at bmsi.com (Stuart D. Gathman) Date: Mon, 6 Jun 2005 10:29:59 -0400 (EDT) Subject: [Pymilter] Re: pymilter vs MIMEDefang / pymilter performances In-Reply-To: Message-ID: On Mon, 6 Jun 2005, stephane Lentz wrote: > For the time being it seems that MIMEDefang has more press and has more > features and history but for basic setups (filtering of subject and/or > banned_exts use) how do pymilter will perform on the memory usage/CPU & > so on points of view ? What about 50.000, 100.000 messages a day set-ups ? The basic engine is very efficient. There is a large ISP that uses it, but they use their own custom milter. > The current TODO list right now is a bit unclear. > I read some notes such as "Check valid domains allowed by internal senders > to detect PCs infected with spam trojans." but could not understand the > idea. Could you clarify the content ? This should be updated to "check outgoing SPF". The idea is that when Windoze PCs become Zombies, they often send out spam email with foreign domains. Checking whether outgoing MAIL FROM domains are legit would catch such spam before it gets into the wild. > Now that the source is on SF, maybe more people could contribute to it ? > I will try to check if I can help someway when I have time left? Get on the mailing list. The current bugs have to do with moving to python2.4 and the email package changes. I need to upload pydspam, since adding tags to a message is seriously broken since upgrading to python2.4. -- Stuart D. Gathman Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154 "Confutatis maledictis, flamis acribus addictis" - background song for a Microsoft sponsored "Where do you want to go from here?" commercial. From stuart at bmsi.com Fri Jun 3 15:27:09 2005 From: stuart at bmsi.com (Stuart D. Gathman) Date: Fri, 3 Jun 2005 15:27:09 -0400 (EDT) Subject: [Pymilter] RE: You are correct, it is a bug (rejecting non-SRS) In-Reply-To: <20050602063825.8C5DA8D8003@2.revivim.org.il> Message-ID: On Thu, 2 Jun 2005, Alex Savguira wrote: > >Since supporting the new email package in python2.4, there are bugs in the > >attachment scanning. Supporting earlier python versions seems to require an > >alternate mime.py implementation. > > So what's the conclusion, does it mean, that because attachment scanning is > buggy, you are not planning such support or that you will wait for the next > release/bug fix ? The problem in 2.4 is that it no longer throws an exception for corrupted rfc822 attachments. Instead, the text of the corrupted attachment is included verbatim without parsing into parts. Unfortunately, this is one of the ways in which virus writers sneak their wares by. Typically, they leave out the terminating boundary, but Outhouse is perfectly happy to tolerate the missing boundary and run the virus code. -- Stuart D. Gathman Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154 "Confutatis maledictis, flamis acribus addictis" - background song for a Microsoft sponsored "Where do you want to go from here?" commercial. From stephane.lentz at gmail.com Sun Jun 5 19:32:46 2005 From: stephane.lentz at gmail.com (stephane Lentz) Date: Mon, 6 Jun 2005 01:32:46 +0200 Subject: [Pymilter] pymilter vs MIMEDefang / pymilter performances Message-ID: Hi Stuart, I've just noticed that you've started to move stuff to SourceForge. Nice. I've some friends who do a lot of filtering against spam at the ruleset local : Block based on the From header or NNTP-Posting-Host or Newsgroups header or the rDNS name or the Subject. I'd like to suggest them to move to Milter solutions. MIMEDefang is of course one option, but pymilter also. Pymilter has way less requirements than mimedefang. I don't remember if I already asked you but did you do a kind of benchmark for Pymilter vs Mimedefang in a similar configuration set-up ? For the time being it seems that MIMEDefang has more press and has more features and history but for basic setups (filtering of subject and/or banned_exts use) how do pymilter will perform on the memory usage/CPU & so on points of view ? What about 50.000, 100.000 messages a day set-ups ? The current TODO list right now is a bit unclear. I read some notes such as "Check valid domains allowed by internal senders to detect PCs infected with spam trojans." but could not understand the idea. Could you clarify the content ? Now that the source is on SF, maybe more people could contribute to it ? I will try to check if I can help someway when I have time left? Regards, SL/ PS: regarding malformed MIME messages, It seems that perl has some advantages with MIME-Tools which has been updated several times by DFS to handle such cases .... -- Stephane.Lentz at gmail.com Ing?nieur S?curit? - ouvert ? des propositions de postes sur Paris ou Bordeaux Comp?tences : SMTP/DNS/FIREWALL/IDS Unix/Linux From stuart at bmsi.com Wed Jun 15 12:39:00 2005 From: stuart at bmsi.com (Stuart D. Gathman) Date: Wed, 15 Jun 2005 12:39:00 -0400 (EDT) Subject: [Pymilter] RE: You are correct, it is a bug (rejecting non-SRS) In-Reply-To: <20050615173515.2EABE8D8009@2.revivim.org.il> Message-ID: On Wed, 15 Jun 2005, Alex Savguira wrote: > I reckon these are exceptions?milter throws when something?goes wrong in > the mime parser class. Do you have any idea how to fix these, what > happends to?these?mail messages? I understand that they are being They get a 451 error - which tells the sender to try again later. Generally, I notice and fix or work around such problems, and then the messages comes through next time the sender tries. > rejected,?would you rather consider accepting them in the case of the > software exception? I would?prefer having a few?junk-mails pass, than a > few business letter being unable to reach the?recepients. Spammers tend to give up after a 451, but real senders will keep trying for several days. I can see your point that an end-user who will be waiting for the next release rather than fixing problems would rather accept messages that trigger an exception. That should be an option. Do you want to add as feature request on sourceforge? I am bouncing this to the mailing list. -- Stuart D. Gathman Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154 "Confutatis maledictis, flamis acribus addictis" - background song for a Microsoft sponsored "Where do you want to go from here?" commercial. From stuart at bmsi.com Wed Jun 15 15:47:21 2005 From: stuart at bmsi.com (Stuart D. Gathman) Date: Wed, 15 Jun 2005 15:47:21 -0400 (EDT) Subject: [Pymilter] RE: You are correct, it is a bug (rejecting non-SRS) In-Reply-To: <20050615173515.2EABE8D8009@2.revivim.org.il> Message-ID: On Wed, 15 Jun 2005, Alex Savguira wrote: > 2005Jun15 19:05:29 [97] CBV: csnooks at saturnee.com > Traceback (most recent call last): > ? File "/usr/lib/python2.4/site-packages/Milter/__init__.py", line 190, > in > ??? milter.set_eom_callback(lambda ctx: ctx.getpriv().eom()) > ? File "bms.py", line 1145, in eom > ??? if len(res) < 3: res += time.time(), > AttributeError: SMTPDataError instance has no attribute '__len__' I've fixed this one in CVS. Module Milter/dsn.py -- Stuart D. Gathman Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154 "Confutatis maledictis, flamis acribus addictis" - background song for a Microsoft sponsored "Where do you want to go from here?" commercial. From stuart at bmsi.com Wed Jun 15 15:56:01 2005 From: stuart at bmsi.com (Stuart D. Gathman) Date: Wed, 15 Jun 2005 15:56:01 -0400 (EDT) Subject: [Pymilter] Header parsing bug In-Reply-To: <20050615173515.2EABE8D8009@2.revivim.org.il> Message-ID: On Wed, 15 Jun 2005, Alex Savguira wrote: > 2005Jun15 18:28:50 [1] connect from 212.116.189.41.knet.co.il at > ('212.116.189.41', 44904) EXTERNAL DYN > 2005Jun15 18:28:50 [1] hello from rmk.co.il > 2005Jun15 18:28:50 [1] mail from () > 2005Jun15 18:28:50 [1] Received-SPF: none (raven.revivim.org.il: > 212.116.189.41 is neither permitted nor denied by domain of rmk.co.il) > 2005Jun15 18:28:50 [1] rcpt to () > 2005Jun15 18:29:21 [1] X-Mailer: Microsoft Outlook Express 6.00.2800.1409 > Traceback (most recent call last): > ? File "/usr/lib/python2.4/site-packages/Milter/__init__.py", line 186, > in > ??? milter.set_header_callback(lambda ctx,fld,val: > ? File "bms.py", line 870, in header > ??? val = parse_header(hval) > ? File "bms.py", line 463, in parse_header > ??? h = decode_header(val) > ? File "/usr/lib/python2.4/email/Header.py", line 93, in decode_header > ??? raise HeaderParseError > email.Errors.HeaderParseError Can you send me or post in a bug report on sourceforge the /var/log/milter/save/*.fail corresponding to this traceback? That would let me reproduce. -- Stuart D. Gathman Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154 "Confutatis maledictis, flamis acribus addictis" - background song for a Microsoft sponsored "Where do you want to go from here?" commercial. From stuart at bmsi.com Thu Jun 16 14:39:35 2005 From: stuart at bmsi.com (Stuart D. Gathman) Date: Thu, 16 Jun 2005 14:39:35 -0400 (EDT) Subject: [Pymilter] RE: Header parsing bug In-Reply-To: <20050616152547.15E968E0006@2.revivim.org.il> Message-ID: I have checked what I think is a fix into CVS on sourceforge. But I still need a test case. I haven't been able to reproduce here. I've fed all kind of garbage to your version of bms.parse_header(), and it handles it all with aplomb. Please send me some mail with a Hebrew subject, which I suspect is what is doing it. -- Stuart D. Gathman Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154 "Confutatis maledictis, flamis acribus addictis" - background song for a Microsoft sponsored "Where do you want to go from here?" commercial. From stuart at bmsi.com Thu Jun 16 23:03:52 2005 From: stuart at bmsi.com (Stuart D. Gathman) Date: Thu, 16 Jun 2005 23:03:52 -0400 (EDT) Subject: [Pymilter] HeaderParse, scan_zip loop, and other bugs In-Reply-To: <20050616152547.15E968E0006@2.revivim.org.il> Message-ID: There were enough important bugs fixed, that I released milter-0.8.1. -- Stuart D. Gathman Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154 "Confutatis maledictis, flamis acribus addictis" - background song for a Microsoft sponsored "Where do you want to go from here?" commercial. From stuart at bmsi.com Wed Jun 22 19:00:34 2005 From: stuart at bmsi.com (Stuart D. Gathman) Date: Wed, 22 Jun 2005 19:00:34 -0400 (EDT) Subject: [Pymilter] SPF API changes Message-ID: Scott Kitterman has joined the Sourceforge pymilter team. He is working on the SPF module. One of the changes is to the SPF result set. New drafts have changed it from none,fail,softfail,neutral,pass,error,unknown to none,fail,softfail,neutral,pass,tempfail,permfail. He also plans to remove the SMTP error code. Is anyone using the spf module independently of pymilter that wants to weigh in on these changes? -- Stuart D. Gathman Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154 "Confutatis maledictis, flamis acribus addictis" - background song for a Microsoft sponsored "Where do you want to go from here?" commercial. From stuart at bmsi.com Fri Jun 24 14:04:42 2005 From: stuart at bmsi.com (Stuart D. Gathman) Date: Fri, 24 Jun 2005 14:04:42 -0400 (EDT) Subject: [Pymilter] pymilter vs MIMEDefang / pymilter performances In-Reply-To: Message-ID: On Mon, 6 Jun 2005, stephane Lentz wrote: > I don't remember if I already asked you but did you do a > kind of benchmark for Pymilter vs Mimedefang in a similar > configuration set-up ? No direct comparison. I'm not out to kill the competition. I would say if you prefer Perl scripting, use MIMEDefang. If Perl code looks like chicken scratches to you, and you love Python, then use pymilter. > For the time being it seems that MIMEDefang has more press and has more > features and history but for basic setups (filtering of subject and/or > banned_exts use) how do pymilter will perform on the memory usage/CPU & > so on points of view ? What about 50.000, 100.000 messages a day set-ups ? As a datapoint, my mail server does 30.000 message per day on a 600Mhz Celeron. It runs dspam as well as SPF and various python policies. It is never less than 96% idle according to sar. However, all but a few hundred of those 30.000 messages are spam which is blocked before ever getting to SMTP DATA. So I can't say what the load would be for 30.000 legitimate messages. One of the pymilter users is an ISP that runs 100.000 plus messages a day using their own milter. So the overhead to get to your python coded milter callback is not much. > I read some notes such as "Check valid domains allowed by internal senders > to detect PCs infected with spam trojans." but could not understand the > idea. Could you clarify the content ? Here is a sample log line: 2005Jun22 12:01:04 [12430] REJECT: zombie PC at 192.168.100.171 sending MAIL FROM debby at fedex.com No, fedex.com does not use pymilter, and there is no one named debby at my client. :-) But the idiot using the PC at 192.168.100.171 has downloaded and installed some stupid weatherbar/hotbar/ aquariumscreensaver that is actually a spam bot. > Now that the source is on SF, maybe more people could contribute to it ? We now have another developer working on SPF. -- Stuart D. Gathman Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154 "Confutatis maledictis, flamis acribus addictis" - background song for a Microsoft sponsored "Where do you want to go from here?" commercial.