From me at tspycher.com  Tue Mar 20 12:22:47 2012
From: me at tspycher.com (Thomas Spycher)
Date: Tue, 20 Mar 2012 17:22:47 +0100
Subject: [Pymilter] Adding Recipient takes extremely long
Message-ID: <CFAC5CE3-8475-4E1C-BFD6-03F61B2E915E@tspycher.com>

Hi

I'm doing some performance testings for a high security appliance based on postfix, ubuntu and PythonMilter.
I've figured out, that adding a recipient takes extremely long.

    def eom(self):
        "Called at the end of message."
         self.addrcpt( "<mail at domain.tld>")

See the attached graphs comparing SMTP Connections with or without milter enabled:



any Ideas why this happens?

Thank you and greetings from Switzerland

Tom

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gathman.org/pipermail/pymilter/attachments/20120320/e1fbf9a9/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Basic Header Change - V4Milter.png
Type: image/png
Size: 53472 bytes
Desc: not available
URL: <http://gathman.org/pipermail/pymilter/attachments/20120320/e1fbf9a9/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ONLYRecpt Rewrites - V4Milter.png
Type: image/png
Size: 50242 bytes
Desc: not available
URL: <http://gathman.org/pipermail/pymilter/attachments/20120320/e1fbf9a9/attachment-0001.png>

From stuart at bmsi.com  Tue Mar 20 15:20:55 2012
From: stuart at bmsi.com (Stuart D Gathman)
Date: Tue, 20 Mar 2012 15:20:55 -0400
Subject: [Pymilter] Adding Recipient takes extremely long
In-Reply-To: <CFAC5CE3-8475-4E1C-BFD6-03F61B2E915E@tspycher.com>
References: <CFAC5CE3-8475-4E1C-BFD6-03F61B2E915E@tspycher.com>
Message-ID: <4F68D897.7080907@bmsi.com>

Long ago, Nostradamus foresaw that on 03/20/2012 12:22 PM, Thomas 
Spycher would write:
> I'm doing some performance testings for a high security appliance 
> based on postfix, ubuntu and PythonMilter.
> I've figured out, that adding a recipient takes extremely long.
>
>     def eom(self):
>         "Called at the end of message."
>          self.addrcpt( "<mail at domain.tld <mailto:mail at domain.tld>>")
As far as I can see, the performance of this is entirely up to the MTA - 
postfix in your case.  Can you try it with sendmail?  Or is there a 
script I could try with sendmail?  If sendmail has more reasonable 
overhead for addrcpt, I would call that a performance bug in postfix.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gathman.org/pipermail/pymilter/attachments/20120320/3f0baacd/attachment.html>

From me at tspycher.com  Wed Mar 21 04:58:46 2012
From: me at tspycher.com (Thomas Spycher)
Date: Wed, 21 Mar 2012 09:58:46 +0100
Subject: [Pymilter] Adding Recipient takes extremely long
In-Reply-To: <4F68D897.7080907@bmsi.com>
References: <CFAC5CE3-8475-4E1C-BFD6-03F61B2E915E@tspycher.com>
	<4F68D897.7080907@bmsi.com>
Message-ID: <FDBFDF94-4F7E-4A5B-BE98-CAAC2D867781@tspycher.com>

Hi Stuart

Thanks for your fast reply. It could be interesting to get some knowledge from milter/postfix insides. How does milter add an recipient to an arriving mail?
Is there a way to manipulate the whole raw header with regexp from milter?

In detail my script is going to replace all recipient addresses (to, cc, bcc) with the same address, but with a uuid4 and a + prefixed. This is due to trace a message trough the whole system and logs, even if it gets requeued.

Thanks for your help

On Mar 20, 2012, at 20:20 , Stuart D Gathman wrote:

> Long ago, Nostradamus foresaw that on 03/20/2012 12:22 PM, Thomas Spycher would write:
>> I'm doing some performance testings for a high security appliance based on postfix, ubuntu and PythonMilter.
>> I've figured out, that adding a recipient takes extremely long.
>> 
>>     def eom(self):
>>         "Called at the end of message."
>>          self.addrcpt( "<mail at domain.tld>")
> As far as I can see, the performance of this is entirely up to the MTA - postfix in your case.  Can you try it with sendmail?  Or is there a script I could try with sendmail?  If sendmail has more reasonable overhead for addrcpt, I would call that a performance bug in postfix.
> _______________________________________________
> Pymilter mailing list
> Pymilter at bmsi.com
> http://www.bmsi.com/mailman/listinfo/pymilter



From stuart at bmsi.com  Wed Mar 21 11:02:53 2012
From: stuart at bmsi.com (Stuart D Gathman)
Date: Wed, 21 Mar 2012 11:02:53 -0400
Subject: [Pymilter] Adding Recipient takes extremely long
In-Reply-To: <FDBFDF94-4F7E-4A5B-BE98-CAAC2D867781@tspycher.com>
References: <CFAC5CE3-8475-4E1C-BFD6-03F61B2E915E@tspycher.com>
	<4F68D897.7080907@bmsi.com>
	<FDBFDF94-4F7E-4A5B-BE98-CAAC2D867781@tspycher.com>
Message-ID: <4F69ED9D.9050604@bmsi.com>

Long ago, Nostradamus foresaw that on 03/21/2012 04:58 AM, Thomas
Spycher would write:
> Thanks for your fast reply. It could be interesting to get some knowledge from milter/postfix insides. How does milter add an recipient to an arriving mail?
> Is there a way to manipulate the whole raw header with regexp from milter?
>
> In detail my script is going to replace all recipient addresses (to, cc, bcc) with the same address, but with a uuid4 and a + prefixed. This is due to trace a message trough the whole system and logs, even if it gets requeued.
>
libmilter doesn't add recipients.  It sends a message to the MTA saying
"please add this recipient: foobar".  The MTA then adds the recipient
and sends back a status code.  There will be overhead in the MTA, like
adding to queues, etc, but not several seconds worth that you are
seeing.  This looks like a postfix bug.  I am not running your
benchmark, but I don't see that kind of large delay in my milter log for
adding recipients.


From stuart at bmsi.com  Wed Mar 21 11:13:25 2012
From: stuart at bmsi.com (Stuart D Gathman)
Date: Wed, 21 Mar 2012 11:13:25 -0400
Subject: [Pymilter] Adding Recipient takes extremely long
In-Reply-To: <FDBFDF94-4F7E-4A5B-BE98-CAAC2D867781@tspycher.com>
References: <CFAC5CE3-8475-4E1C-BFD6-03F61B2E915E@tspycher.com>
	<4F68D897.7080907@bmsi.com>
	<FDBFDF94-4F7E-4A5B-BE98-CAAC2D867781@tspycher.com>
Message-ID: <4F69F015.3000602@bmsi.com>

Long ago, Nostradamus foresaw that on 03/21/2012 04:58 AM, Thomas
Spycher would write:
> In detail my script is going to replace all recipient addresses (to, cc, bcc) with the same address, but with a uuid4 and a + prefixed. This is due to trace a message trough the whole system and logs, even if it gets requeued.
>
Looking at the postfix documentation, non-smtp milters cannot change
recipients.  Is this an smtp time milter you are using?