From LWapnitsky at ph.wrtdesign.com Tue May 22 11:54:11 2012 From: LWapnitsky at ph.wrtdesign.com (Larry G. Wapnitsky) Date: Tue, 22 May 2012 11:54:11 -0400 Subject: [Pymilter] Unknown command and further processing Message-ID: I have the following code in a milter that determines when Postfix has received an unknown command: @Milter.noreply def unknown(self, cmd): self.unkMilterlog.warning('Invalid command sent: %s' % cmd) try: self.unkMilterlog.warning('From: %s' % self.canon_from) except: self.unkMilterlog.warning('unable to determine Sender') try: for R in self.R: for recipient in R: if not len(recipient) < 1: self.log('To %s' % recipient) except: self.unkMilterlog.warning('unable to determine Recipient(s)') return Milter.CONTINUE Is it possible to do further processing on a message when "unknown" has been called? Part of the reason this was initially created (for logging purposes) was that another milter was not being hit when an unknown command was being interpreted. What I'd like to do is take the full message and put it into a MySQL blob for further analysis. Thanks, Larry ----wrtmail--%3423wrt%---- -------------- next part -------------- An HTML attachment was scrubbed... URL: From stuart at bmsi.com Tue May 22 14:59:06 2012 From: stuart at bmsi.com (Stuart D Gathman) Date: Tue, 22 May 2012 14:59:06 -0400 Subject: [Pymilter] Unknown command and further processing In-Reply-To: References: Message-ID: <4FBBE1FA.5060100@bmsi.com> Long ago, Nostradamus foresaw that on 05/22/2012 11:54 AM, Larry G. Wapnitsky would write: > > I have the following code in a milter that determines when Postfix has > received an unknown command: > > @Milter.noreply > > def unknown(self, cmd): > > self.unkMilterlog.warning('Invalid command sent: %s' % cmd) > > try: > > self.unkMilterlog.warning('From: %s' % self.canon_from) > > except: > > self.unkMilterlog.warning('unable to determine Sender') > > try: > > for R in self.R: > > for recipient in R: > > if not len(recipient) < 1: self.log('To %s' % > recipient) > > except: > > self.unkMilterlog.warning('unable to determine Recipient(s)') > > return Milter.CONTINUE > > Is it possible to do further processing on a message when "unknown" > has been called? Part of the reason this was initially created (for > logging purposes) was that another milter was not being hit when an > unknown command was being interpreted. What I'd like to do is take > the full message and put it into a MySQL blob for further analysis. > Nothing about milter prevents further processing. I tested with sendmail, results below. What problem are you having, exactly? Milter code: @Milter.noreply def unknown(self, cmd): self.log('Invalid command sent: %s' % cmd) return Milter.CONTINUE Client side: 220 mail.bmsi.com ESMTP Sendmail 8.14.3/8.14.3; Tue, 22 May 2012 14:50:40 -0400 HELO hulk 250 mail.bmsi.com Hello hulk [192.168.0.102], pleased to meet you HELP 214-2.0.0 This is sendmail version 8.14.3 214-2.0.0 Topics: 214-2.0.0 HELO EHLO MAIL RCPT DATA 214-2.0.0 RSET NOOP QUIT HELP VRFY 214-2.0.0 EXPN VERB ETRN DSN AUTH 214-2.0.0 STARTTLS 214-2.0.0 For more info use "HELP ". 214-2.0.0 To report bugs in the implementation see 214-2.0.0 http://www.sendmail.org/email-addresses.html 214-2.0.0 For local information send email to Postmaster at your site. 214 2.0.0 End of HELP info PING super 500 5.5.1 Command unrecognized: "PING super" MAIL FROM: 250 2.1.0 ... Sender ok RCPT TO: 250 2.1.5 ... Recipient ok SCRIBBLE: foo 500 5.5.1 Command unrecognized: "SCRIBBLE: foo" DATA 354 Enter mail, end with "." on a line by itself Testing . 250 2.0.0 q4MIoefa002370 Message accepted for delivery quit 221 2.0.0 mail.bmsi.com closing connection Connection closed by foreign host. Milter log: 2012May22 14:50:40 [1] connect from hulk at ('192.168.0.102', 34367) INTERNAL 2012May22 14:50:54 [1] hello from hulk 2012May22 14:51:14 [1] Invalid command sent: PING super 2012May22 14:51:49 [1] mail from () 2012May22 14:51:58 [1] rcpt to () 2012May22 14:52:06 [1] Invalid command sent: SCRIBBLE: foo 2012May22 14:52:17 [1] Authentication-Results: mail.bmsi.com; iprev=pass policy.iprev=192.168.0.102 (hulk) 2012May22 14:52:17 [1] X-DSpam-Score: 0.152536 2012May22 14:52:17 [1] eom -------------- next part -------------- An HTML attachment was scrubbed... URL: From LWapnitsky at ph.wrtdesign.com Wed May 23 12:01:29 2012 From: LWapnitsky at ph.wrtdesign.com (Larry G. Wapnitsky) Date: Wed, 23 May 2012 12:01:29 -0400 Subject: [Pymilter] Unknown command and further processing Message-ID: No problem yet. I'm modifying two bits of existing code write now and wanted to make sure I could do what I needed to before continuing on. I'll keep you posted on progress. Larry G. Wapnitsky MBA, MCSE, MCP+I IT SUPPORT & ADMINISTRATION COORDINATOR WRT | Wallace Roberts & Todd, LLC 1700 Market Street, 28th Fl Philadelphia, PA 19103 T 215.430.5068 C 215.713.8635 E lwapnitsky at WRTdesign.com From: Stuart D Gathman [mailto:stuart at bmsi.com] Sent: Tuesday, May 22, 2012 2:59 PM To: pymilter at bmsi.com Subject: Re: [Pymilter] Unknown command and further processing Long ago, Nostradamus foresaw that on 05/22/2012 11:54 AM, Larry G. Wapnitsky would write: I have the following code in a milter that determines when Postfix has received an unknown command: @Milter.noreply def unknown(self, cmd): self.unkMilterlog.warning('Invalid command sent: %s' % cmd) try: self.unkMilterlog.warning('From: %s' % self.canon_from) except: self.unkMilterlog.warning('unable to determine Sender') try: for R in self.R: for recipient in R: if not len(recipient) < 1: self.log('To %s' % recipient) except: self.unkMilterlog.warning('unable to determine Recipient(s)') return Milter.CONTINUE Is it possible to do further processing on a message when "unknown" has been called? Part of the reason this was initially created (for logging purposes) was that another milter was not being hit when an unknown command was being interpreted. What I'd like to do is take the full message and put it into a MySQL blob for further analysis. Nothing about milter prevents further processing. I tested with sendmail, results below. What problem are you having, exactly? Milter code: @Milter.noreply def unknown(self, cmd): self.log('Invalid command sent: %s' % cmd) return Milter.CONTINUE Client side: 220 mail.bmsi.com ESMTP Sendmail 8.14.3/8.14.3; Tue, 22 May 2012 14:50:40 -0400 HELO hulk 250 mail.bmsi.com Hello hulk [192.168.0.102], pleased to meet you HELP 214-2.0.0 This is sendmail version 8.14.3 214-2.0.0 Topics: 214-2.0.0 HELO EHLO MAIL RCPT DATA 214-2.0.0 RSET NOOP QUIT HELP VRFY 214-2.0.0 EXPN VERB ETRN DSN AUTH 214-2.0.0 STARTTLS 214-2.0.0 For more info use "HELP ". 214-2.0.0 To report bugs in the implementation see 214-2.0.0 http://www.sendmail.org/email-addresses.html 214-2.0.0 For local information send email to Postmaster at your site. 214 2.0.0 End of HELP info PING super 500 5.5.1 Command unrecognized: "PING super" MAIL FROM: 250 2.1.0 ... Sender ok RCPT TO: 250 2.1.5 ... Recipient ok SCRIBBLE: foo 500 5.5.1 Command unrecognized: "SCRIBBLE: foo" DATA 354 Enter mail, end with "." on a line by itself Testing . 250 2.0.0 q4MIoefa002370 Message accepted for delivery quit 221 2.0.0 mail.bmsi.com closing connection Connection closed by foreign host. Milter log: 2012May22 14:50:40 [1] connect from hulk at ('192.168.0.102', 34367) INTERNAL 2012May22 14:50:54 [1] hello from hulk 2012May22 14:51:14 [1] Invalid command sent: PING super 2012May22 14:51:49 [1] mail from () 2012May22 14:51:58 [1] rcpt to () 2012May22 14:52:06 [1] Invalid command sent: SCRIBBLE: foo 2012May22 14:52:17 [1] Authentication-Results: mail.bmsi.com; iprev=pass policy.iprev=192.168.0.102 (hulk) 2012May22 14:52:17 [1] X-DSpam-Score: 0.152536 2012May22 14:52:17 [1] eom ----wrtmail--%3423wrt%---- -------------- next part -------------- An HTML attachment was scrubbed... URL: From LWapnitsky at ph.wrtdesign.com Wed May 23 15:12:24 2012 From: LWapnitsky at ph.wrtdesign.com (Larry G. Wapnitsky) Date: Wed, 23 May 2012 15:12:24 -0400 Subject: [Pymilter] Unknown command and further processing In-Reply-To: References: Message-ID: OK, so here's the issue. My original milter would "crap out" when an unknown command was sent to Postfix. That milter is designed to remove attachments from a message, save them and replace them with a single HTML file pointing to where the attachments have been saved. I was never able to diagnose why until I saw the postfix logs having Unknown commands at the exact same time as the message was received. I started logging these and, lo and behold, coincidence. Now, I have a milter that comes before my original that just passes on e-mail with unknown commands, but logs when they occur, to whom they were sent, etc. I've just written yet another milter (evolved from the original) that I want to call when the unknown commands are hit. This one would log the headers, et. al. to a database, with the attachments as blobs. My concern is that this one will fail as well. How do I tell my milter to only to the DB stuff if the unknown command is received? Should that be a true/false value as the trigger when unknown is hit? Let me know if you'd like to see the code to my milters for clarification. Regards, Larry G. Wapnitsky MBA, MCSE, MCP+I IT SUPPORT & ADMINISTRATION COORDINATOR WRT | Wallace Roberts & Todd, LLC 1700 Market Street, 28th Fl Philadelphia, PA 19103 T 215.430.5068 C 215.713.8635 E lwapnitsky at WRTdesign.com From: Larry G. Wapnitsky Sent: Wednesday, May 23, 2012 12:01 PM To: 'Stuart D Gathman'; pymilter at bmsi.com Subject: RE: [Pymilter] Unknown command and further processing No problem yet. I'm modifying two bits of existing code write now and wanted to make sure I could do what I needed to before continuing on. I'll keep you posted on progress. Larry G. Wapnitsky MBA, MCSE, MCP+I IT SUPPORT & ADMINISTRATION COORDINATOR WRT | Wallace Roberts & Todd, LLC 1700 Market Street, 28th Fl Philadelphia, PA 19103 T 215.430.5068 C 215.713.8635 E lwapnitsky at WRTdesign.com From: Stuart D Gathman [mailto:stuart at bmsi.com] Sent: Tuesday, May 22, 2012 2:59 PM To: pymilter at bmsi.com Subject: Re: [Pymilter] Unknown command and further processing Long ago, Nostradamus foresaw that on 05/22/2012 11:54 AM, Larry G. Wapnitsky would write: I have the following code in a milter that determines when Postfix has received an unknown command: @Milter.noreply def unknown(self, cmd): self.unkMilterlog.warning('Invalid command sent: %s' % cmd) try: self.unkMilterlog.warning('From: %s' % self.canon_from) except: self.unkMilterlog.warning('unable to determine Sender') try: for R in self.R: for recipient in R: if not len(recipient) < 1: self.log('To %s' % recipient) except: self.unkMilterlog.warning('unable to determine Recipient(s)') return Milter.CONTINUE Is it possible to do further processing on a message when "unknown" has been called? Part of the reason this was initially created (for logging purposes) was that another milter was not being hit when an unknown command was being interpreted. What I'd like to do is take the full message and put it into a MySQL blob for further analysis. Nothing about milter prevents further processing. I tested with sendmail, results below. What problem are you having, exactly? Milter code: @Milter.noreply def unknown(self, cmd): self.log('Invalid command sent: %s' % cmd) return Milter.CONTINUE Client side: 220 mail.bmsi.com ESMTP Sendmail 8.14.3/8.14.3; Tue, 22 May 2012 14:50:40 -0400 HELO hulk 250 mail.bmsi.com Hello hulk [192.168.0.102], pleased to meet you HELP 214-2.0.0 This is sendmail version 8.14.3 214-2.0.0 Topics: 214-2.0.0 HELO EHLO MAIL RCPT DATA 214-2.0.0 RSET NOOP QUIT HELP VRFY 214-2.0.0 EXPN VERB ETRN DSN AUTH 214-2.0.0 STARTTLS 214-2.0.0 For more info use "HELP ". 214-2.0.0 To report bugs in the implementation see 214-2.0.0 http://www.sendmail.org/email-addresses.html 214-2.0.0 For local information send email to Postmaster at your site. 214 2.0.0 End of HELP info PING super 500 5.5.1 Command unrecognized: "PING super" MAIL FROM: 250 2.1.0 ... Sender ok RCPT TO: 250 2.1.5 ... Recipient ok SCRIBBLE: foo 500 5.5.1 Command unrecognized: "SCRIBBLE: foo" DATA 354 Enter mail, end with "." on a line by itself Testing . 250 2.0.0 q4MIoefa002370 Message accepted for delivery quit 221 2.0.0 mail.bmsi.com closing connection Connection closed by foreign host. Milter log: 2012May22 14:50:40 [1] connect from hulk at ('192.168.0.102', 34367) INTERNAL 2012May22 14:50:54 [1] hello from hulk 2012May22 14:51:14 [1] Invalid command sent: PING super 2012May22 14:51:49 [1] mail from () 2012May22 14:51:58 [1] rcpt to () 2012May22 14:52:06 [1] Invalid command sent: SCRIBBLE: foo 2012May22 14:52:17 [1] Authentication-Results: mail.bmsi.com; iprev=pass policy.iprev=192.168.0.102 (hulk) 2012May22 14:52:17 [1] X-DSpam-Score: 0.152536 2012May22 14:52:17 [1] eom ----wrtmail--%3423wrt%---- -------------- next part -------------- An HTML attachment was scrubbed... URL: From stuart at bmsi.com Wed May 23 15:17:56 2012 From: stuart at bmsi.com (Stuart D Gathman) Date: Wed, 23 May 2012 15:17:56 -0400 Subject: [Pymilter] Unknown command and further processing In-Reply-To: References: Message-ID: <4FBD37E4.50901@bmsi.com> On 05/23/2012 03:12 PM, Larry G. Wapnitsky expounded in part: > > OK, so here's the issue. > > My original milter would "crap out" when an unknown command was sent > to Postfix. That milter is designed to remove attachments from a > message, save them and replace them with a single HTML file pointing > to where the attachments have been saved. I was never able to > diagnose why until I saw the postfix logs having Unknown commands at > the exact same time as the message was received. I started logging > these and, lo and behold, coincidence. > > Now, I have a milter that comes before my original that just passes on > e-mail with unknown commands, but logs when they occur, to whom they > were sent, etc. > > I've just written yet another milter (evolved from the original) that > I want to call when the unknown commands are hit. This one would log > the headers, et. al. to a database, with the attachments as blobs. > > My concern is that this one will fail as well. How do I tell my > milter to only to the DB stuff if the unknown command is received? > Should that be a true/false value as the trigger when unknown is hit? > > Let me know if you'd like to see the code to my milters for clarification. > > Sure, just set a flag in the unknown callback, and log headers and body only when hitting that. You could also just set a flag in your original milter, and avoid rejecting the message in later callbacks. It is not clear that you have verified whether postfix does indeed go on processing the mail after seeing an unknown command. -------------- next part -------------- An HTML attachment was scrubbed... URL: From LWapnitsky at ph.wrtdesign.com Wed May 23 15:20:23 2012 From: LWapnitsky at ph.wrtdesign.com (Larry G. Wapnitsky) Date: Wed, 23 May 2012 15:20:23 -0400 Subject: [Pymilter] Unknown command and further processing In-Reply-To: <4FBD37E4.50901@bmsi.com> References: <4FBD37E4.50901@bmsi.com> Message-ID: It does continue on, but since there are invalid parts to the message, it only sends the body, not the attachments, nor any modified attachments that the initial milter added. Larry G. Wapnitsky MBA, MCSE, MCP+I IT SUPPORT & ADMINISTRATION COORDINATOR WRT | Wallace Roberts & Todd, LLC 1700 Market Street, 28th Fl Philadelphia, PA 19103 T 215.430.5068 C 215.713.8635 E lwapnitsky at WRTdesign.com From: Stuart D Gathman [mailto:stuart at bmsi.com] Sent: Wednesday, May 23, 2012 3:18 PM To: Larry G. Wapnitsky Cc: pymilter at bmsi.com Subject: Re: [Pymilter] Unknown command and further processing On 05/23/2012 03:12 PM, Larry G. Wapnitsky expounded in part: OK, so here's the issue. My original milter would "crap out" when an unknown command was sent to Postfix. That milter is designed to remove attachments from a message, save them and replace them with a single HTML file pointing to where the attachments have been saved. I was never able to diagnose why until I saw the postfix logs having Unknown commands at the exact same time as the message was received. I started logging these and, lo and behold, coincidence. Now, I have a milter that comes before my original that just passes on e-mail with unknown commands, but logs when they occur, to whom they were sent, etc. I've just written yet another milter (evolved from the original) that I want to call when the unknown commands are hit. This one would log the headers, et. al. to a database, with the attachments as blobs. My concern is that this one will fail as well. How do I tell my milter to only to the DB stuff if the unknown command is received? Should that be a true/false value as the trigger when unknown is hit? Let me know if you'd like to see the code to my milters for clarification. Sure, just set a flag in the unknown callback, and log headers and body only when hitting that. You could also just set a flag in your original milter, and avoid rejecting the message in later callbacks. It is not clear that you have verified whether postfix does indeed go on processing the mail after seeing an unknown command. ----wrtmail--%3423wrt%---- -------------- next part -------------- An HTML attachment was scrubbed... URL: From LWapnitsky at ph.wrtdesign.com Wed May 23 15:23:26 2012 From: LWapnitsky at ph.wrtdesign.com (Larry G. Wapnitsky) Date: Wed, 23 May 2012 15:23:26 -0400 Subject: [Pymilter] Unknown command and further processing In-Reply-To: <4FBD37E4.50901@bmsi.com> References: <4FBD37E4.50901@bmsi.com> Message-ID: Also, I understand how to set the flag, but how do I control the header, etc. reading? What's the order that the commands are usually processed in, and should I remove Milter.noreply from header reading, etc.? Thanks Larry G. Wapnitsky MBA, MCSE, MCP+I IT SUPPORT & ADMINISTRATION COORDINATOR WRT | Wallace Roberts & Todd, LLC 1700 Market Street, 28th Fl Philadelphia, PA 19103 T 215.430.5068 C 215.713.8635 E lwapnitsky at WRTdesign.com From: Stuart D Gathman [mailto:stuart at bmsi.com] Sent: Wednesday, May 23, 2012 3:18 PM To: Larry G. Wapnitsky Cc: pymilter at bmsi.com Subject: Re: [Pymilter] Unknown command and further processing On 05/23/2012 03:12 PM, Larry G. Wapnitsky expounded in part: OK, so here's the issue. My original milter would "crap out" when an unknown command was sent to Postfix. That milter is designed to remove attachments from a message, save them and replace them with a single HTML file pointing to where the attachments have been saved. I was never able to diagnose why until I saw the postfix logs having Unknown commands at the exact same time as the message was received. I started logging these and, lo and behold, coincidence. Now, I have a milter that comes before my original that just passes on e-mail with unknown commands, but logs when they occur, to whom they were sent, etc. I've just written yet another milter (evolved from the original) that I want to call when the unknown commands are hit. This one would log the headers, et. al. to a database, with the attachments as blobs. My concern is that this one will fail as well. How do I tell my milter to only to the DB stuff if the unknown command is received? Should that be a true/false value as the trigger when unknown is hit? Let me know if you'd like to see the code to my milters for clarification. Sure, just set a flag in the unknown callback, and log headers and body only when hitting that. You could also just set a flag in your original milter, and avoid rejecting the message in later callbacks. It is not clear that you have verified whether postfix does indeed go on processing the mail after seeing an unknown command. ----wrtmail--%3423wrt%---- -------------- next part -------------- An HTML attachment was scrubbed... URL: From stuart at bmsi.com Wed May 23 15:45:23 2012 From: stuart at bmsi.com (Stuart D Gathman) Date: Wed, 23 May 2012 15:45:23 -0400 Subject: [Pymilter] Unknown command and further processing In-Reply-To: References: <4FBD37E4.50901@bmsi.com> Message-ID: <4FBD3E53.6070501@bmsi.com> On 05/23/2012 03:23 PM, Larry G. Wapnitsky expounded in part: > > Also, I understand how to set the flag, but how do I control the > header, etc. reading? What's the order that the commands are usually > processed in, and should I remove Milter.noreply from header reading, > etc.? > There is no point in removing Miler.noreply unless you ever return something other than Milter.CONTINUE. SMTP protocol defines the order of commands (and therefore callbacks). Any deviation causes the client to be rejected. Here is a summary: connect helo ---- repeat for each message ----- envfrom envrcpt ... envrcpt --- repeat for each rcpt data header ... header --- repeat for each header eoh body ... body --- repeat for each body chunk eom ------------------------------------------- close For instance, (untested, from the top of my head): @Milter.noreply def envfrom(self,from,args): self.unknown_seen = False self.fp = None ... return Milter.CONTINUE @Milter.noreply def unknown(self,cmd): self.log('Unknown:',cmd) self.unknown_seen = True self.fp = StringIO.StringIO() return Milter.CONTINUE @Milter.noreply def header(self,name,val): if self.unknown_seen: self.fp.write("%s: %s\n" % (name,val)) return Milter.CONTINUE @Milter.noreply def eoh(self): if self.unknown_seen: self.fp.write('\n') return Milter.CONTINUE @Milter.noreply def body(self,buf): if self.unknown_seen: self.fp.write(buf) return Milter.CONTINUE @Milter.noreply def eom(self): if self.unknown_seen: self.fp.seek(0) self.store_in_database(self.fp.getvalue()) return Milter.CONTINUE -------------- next part -------------- An HTML attachment was scrubbed... URL: From stuart at bmsi.com Wed May 23 15:45:23 2012 From: stuart at bmsi.com (Stuart D Gathman) Date: Wed, 23 May 2012 15:45:23 -0400 Subject: [Pymilter] Unknown command and further processing In-Reply-To: References: <4FBD37E4.50901@bmsi.com> Message-ID: <4FBD3E53.6070501@bmsi.com> On 05/23/2012 03:23 PM, Larry G. Wapnitsky expounded in part: > > Also, I understand how to set the flag, but how do I control the > header, etc. reading? What's the order that the commands are usually > processed in, and should I remove Milter.noreply from header reading, > etc.? > There is no point in removing Miler.noreply unless you ever return something other than Milter.CONTINUE. SMTP protocol defines the order of commands (and therefore callbacks). Any deviation causes the client to be rejected. Here is a summary: connect helo ---- repeat for each message ----- envfrom envrcpt ... envrcpt --- repeat for each rcpt data header ... header --- repeat for each header eoh body ... body --- repeat for each body chunk eom ------------------------------------------- close For instance, (untested, from the top of my head): @Milter.noreply def envfrom(self,from,args): self.unknown_seen = False self.fp = None ... return Milter.CONTINUE @Milter.noreply def unknown(self,cmd): self.log('Unknown:',cmd) self.unknown_seen = True self.fp = StringIO.StringIO() return Milter.CONTINUE @Milter.noreply def header(self,name,val): if self.unknown_seen: self.fp.write("%s: %s\n" % (name,val)) return Milter.CONTINUE @Milter.noreply def eoh(self): if self.unknown_seen: self.fp.write('\n') return Milter.CONTINUE @Milter.noreply def body(self,buf): if self.unknown_seen: self.fp.write(buf) return Milter.CONTINUE @Milter.noreply def eom(self): if self.unknown_seen: self.fp.seek(0) self.store_in_database(self.fp.getvalue()) return Milter.CONTINUE -------------- next part -------------- An HTML attachment was scrubbed... URL: