From tlyons at ivenue.com Mon Mar 1 18:28:55 2010 From: tlyons at ivenue.com (Todd Lyons) Date: Mon, 1 Mar 2010 15:28:55 -0800 Subject: [Pymilter] Maybe seperate peer queries? Message-ID: <48b1344b1003011528j67e46d56ib831076e564ae3d7@mail.gmail.com> One of the things that kinda bugs me about peer queries is that a new entry gets pushed onto the circular queue. What do you think about a new query type called P which does the same as a Q query except don't create a new UMIS, so it doesnt push it onto the circular queue. -- Regards... Todd I seek the truth...it is only persistence in self-delusion and ignorance that does harm. -- Marcus Aurealius From stuart at bmsi.com Mon Mar 1 23:27:04 2010 From: stuart at bmsi.com (Stuart D. Gathman) Date: Mon, 1 Mar 2010 23:27:04 -0500 (EST) Subject: [Pymilter] Maybe seperate peer queries? In-Reply-To: <48b1344b1003011528j67e46d56ib831076e564ae3d7@mail.gmail.com> References: <48b1344b1003011528j67e46d56ib831076e564ae3d7@mail.gmail.com> Message-ID: On Mon, 1 Mar 2010, Todd Lyons wrote: > One of the things that kinda bugs me about peer queries is that a new > entry gets pushed onto the circular queue. What do you think about a > new query type called P which does the same as a Q query except don't > create a new UMIS, so it doesnt push it onto the circular queue. That could be useful. That would prevent voting later, but in many situations, it is purely a query. Another way would be an empty or otherwise trivial UMIS to prevent adding to the queue. -- Stuart D. Gathman Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154 "Confutatis maledictis, flammis acribus addictis" - background song for a Microsoft sponsored "Where do you want to go from here?" commercial. From tlyons at ivenue.com Tue Mar 2 11:54:35 2010 From: tlyons at ivenue.com (Todd Lyons) Date: Tue, 2 Mar 2010 08:54:35 -0800 Subject: [Pymilter] Maybe seperate peer queries? In-Reply-To: References: <48b1344b1003011528j67e46d56ib831076e564ae3d7@mail.gmail.com> Message-ID: <48b1344b1003020854h5b46752am5cf012a8a5da2875@mail.gmail.com> On Mon, Mar 1, 2010 at 8:27 PM, Stuart D. Gathman wrote: >> What do you think about a >> new query type called P which does the same as a Q query except don't >> create a new UMIS, so it doesnt push it onto the circular queue. > That could be useful. ?That would prevent voting later, but in many > situations, it is purely a query. ?Another way would be an empty or > otherwise trivial UMIS to prevent adding to the queue. I think I like that concept a lot better. Maybe make a provision in a query such that if the UMIS is less than X characters, it's treated as a pure informational query, not as something that needs to be tracked. Additionally, the result returned could be altered to present more detail. I like to see the floating point numbers instead of just the integers. I considered altering the message header to do that, but I figure you obfuscate that for a reason. OT: So far my two node gossip system has been running for a little over an hour on a two server exim mail system. I'm generating feedback solely on the spamassassin score: less than 0 points is -1 (ham), 0-4.9 points is 0 (neutral), and more than 5 points is 1 (spam). Thus far I've seen on one gossip server: 283 neutral (31.8%) 539 ham (60.6%) 66 spam (7.4%) This is _after_ all RBL checks and header checks. To put it in perspective, the gossip server generated 1280 PREPEND responses, but of those only 888 actually made it past header checks to the point of getting scanned for spam and sending a feedback response, so I'm rejecting about 31% of email that made it past my RBL checks, valid user checks, etc, due to malformed headers, viruses, phishing, etc (but before spam checks). That seems high though, so I might be misinterpreting some data somewhere. -- Regards... Todd I seek the truth...it is only persistence in self-delusion and ignorance that does harm. -- Marcus Aurealius From tlyons at ivenue.com Tue Mar 2 12:58:53 2010 From: tlyons at ivenue.com (Todd Lyons) Date: Tue, 2 Mar 2010 09:58:53 -0800 Subject: [Pymilter] general gossip questions In-Reply-To: References: <48b1344b1002221153ub89ad40jdb6ee34b973b00ba@mail.gmail.com> <48b1344b1002251516t67d53f97p1199b7593d4480c6@mail.gmail.com> Message-ID: <48b1344b1003020958m1fd20ceav28d2a8e606d83a5d@mail.gmail.com> On Fri, Feb 26, 2010 at 12:21 PM, Stuart D. Gathman wrote: > >> I was able to telnet multiple times to the daemon in my testing and >> never had problems with concurrency, so I'm not sure that I even >> comprehend what the REUSE_ADDR issue actually is. > > Telnet to daemon. ?Restart daemon with session still active (something > that will almost always be the case in production). > > Daemon will shutdown for restart, but won't start again for 5 mins. > Error in log is "socket in use". ?The SO_REUSE_ADDR socket option is > supposed to let you immediately reuse the socket without trying to > shutdown active connections. > > Somehow, the allow_reuse_address flag doesn't actually result in the > socket option getting set. ?I have spent a little bit of time debugging the > TCPServer python code, and can't see where it goes wrong. I have some more detail on this, but still no solution yet. Maybe this feedback will spark something in your thought process, or maybe it will just be recounting what you've already seen on your own system. Recall that I setup a two node system. The mail servers are named ivwm51 and ivwm52, each with pygossip running and configured as peers of each other. * The exim on ivwm51 is configured to talk to pygossip on ivwm51, and exim on ivwm52 is configured to talk to pygossip on ivwm52. I see messages in both logs reflecting peer responses and reputation. No problem. Traffic is flowing at a couple messages per second on each machine, not an overly heavy load. Now I restart pygossip on ivwm51. It gets the error you speak of. At this point, exim on ivwm51 can query the pygossip on ivwm51, pygossip on ivwm52 can query the peer on ivwm52, but the pygossip on ivwm52 cannot query the peer on ivwm51. Ok, control time: I stop and start pygossip on ivwm52. Now ivwm52 can get peer information from ivwm51 again, but ivwm51 can no longer get peer information from ivwm52. The only way to fix it is to stop both of them and start them both simultaneously. I have a theory about this. The big red flag for me is that if peer 1 is restarted, peer 2 can no longer get info from it. Restarting both simultaneously works properly, and the act of opening port 11900 before any client calls to peers are initiated seems to be a contributing factor whether it will succeed or not. I hope this description makes some sense. * Side question: For the peer list, you don't include the pygossip running at localhost as one of the peers, do you? On ivwm51 I put ivwm52 as a peer, and on ivwm52 I put ivwm51 as a peer. * Second side question: Do I increment the TTL for each peer that I add to the list? If I added a third pygossip server ivwm53, would I need to increment the TTL? It _seems_ to me like I would not have to (i.e. it seems like a hop count), but I'm not 100% sure I'm reading the code accurately. >> I plan to put pygossip on two servers, configure them as peers, and >> set my TTL to 2. ?Then I've got 8 servers that I will point at those I have the TTL set to 1 not 2. > Note that Gossip is compatible with "AOL style" user feedback as well > (where user complaints about an email form a reputation that will eventually > get a sender kicked off AOL. ?AOL tracks IPs rather than domains). I already extract IP information for an internal RBL from ARF formatted feedback emails, so I don't think it would be too difficult to add another subroutine to create a UMIS and then submit spam feedback for it during the ARF processing. That's a great idea, thanks! Now I'm working on the milter (sorry, but I prefer perl :-), will let you know how it works out. -- Regards... Todd I seek the truth...it is only persistence in self-delusion and ignorance that does harm. -- Marcus Aurealius From tlyons at ivenue.com Tue Mar 2 16:35:29 2010 From: tlyons at ivenue.com (Todd Lyons) Date: Tue, 2 Mar 2010 13:35:29 -0800 Subject: [Pymilter] Too harsh? Message-ID: <48b1344b1003021335h2a8d1f02m138a61c70c7e375@mail.gmail.com> One of the things I am doing with my ruleset is making it so that you can override the feedback score. The goal of this is to expand the scope of the reputation to more than just the spam score. Why? When the email is about to get rejected for some gross transgression before it would normally do the spam scan, I still want affect the reputation negatively because of it. In that light, do you think any of the following is too harsh? 1. If an email comes in to a non-existent recipient (but valid local domain), set feedback to1 for that UMIS. 2. If an email comes in that is not a valid local recipient (i.e. someone trying to see if you're an open relay), set feedback to 1 for that UMIS. 3. Various header checks (subject longer than 998 characters, no Date header), set feedback to 1 for that UMIS. In my case, I reject all of these any way. Limiting or slowing future attempts to "do bad things" doesn't seem to bad to me. Of course, if the sending user is authenticated, none of these apply. -- Regards... Todd I seek the truth...it is only persistence in self-delusion and ignorance that does harm. -- Marcus Aurealius From stuart at bmsi.com Thu Mar 4 19:03:10 2010 From: stuart at bmsi.com (Stuart D. Gathman) Date: Thu, 4 Mar 2010 19:03:10 -0500 (EST) Subject: [Pymilter] Too harsh? In-Reply-To: <48b1344b1003021335h2a8d1f02m138a61c70c7e375@mail.gmail.com> References: <48b1344b1003021335h2a8d1f02m138a61c70c7e375@mail.gmail.com> Message-ID: On Tue, 2 Mar 2010, Todd Lyons wrote: > negatively because of it. In that light, do you think any of the > following is too harsh? If you follow the example of using authentication status as a qualifier (pass,neutral,dkim,etc), then there is no downside. If example.com:neutral hits your triggers and gets banned, that won't block example.com:pass or example.com:guess (passes best_guess). The important feature that I am working on is having a web page URL will a full explanation referenced in rejections, like AOL does. I offer fairly complete multi-line rejection explanations on SMTP, but most brain dead email systems (the ones that are likely to get rejected) throw all that away and give the end user something useless. If they do pass it through, the end user ignore it. So pretty much only one line will get to the end user. Some systems give them the first line, some give them the last line. Hence this is why AOL sends exactly one line with an URL. -- Stuart D. Gathman Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154 "Confutatis maledictis, flammis acribus addictis" - background song for a Microsoft sponsored "Where do you want to go from here?" commercial. From tlyons at ivenue.com Sat Mar 6 11:08:27 2010 From: tlyons at ivenue.com (Todd Lyons) Date: Sat, 6 Mar 2010 08:08:27 -0800 Subject: [Pymilter] Too harsh? In-Reply-To: References: <48b1344b1003021335h2a8d1f02m138a61c70c7e375@mail.gmail.com> Message-ID: <48b1344b1003060808p4ef0450au2b1f06650d42c123@mail.gmail.com> On Thu, Mar 4, 2010 at 4:03 PM, Stuart D. Gathman wrote: > > The important feature that I am working on is having a web page URL > will a full explanation referenced in rejections, like AOL does. Ours is at http://postmaster.ivenue.com. However it does not reference the newly implemented sender reputation system. It's only running on the new system anyway, we have an older sendmail based system and a new exim based system. The exim version was easy to code up. The miltered version is not so easy. You probably will chuckle when you read this next sentence: I had forgotten that a milter cannot see headers that other milters have added. So any attempt to create a reputation milter that does not also handle the spam scoring will have to just log profusely and rely on some external process to parse the logs and submit the feedback. I will probably use swatch for that, or maybe write a simple helper daemon that does pretty much what swatch does. I haven't thought too much on that just yet. > So pretty much only one line will get to the end user. ?Some systems My rejection message is pretty straightforward: Sender reputation poor, retry sending later 282bf15886012068508968657d08940e,-76,1 But when I am in debug mode checking things out, that message wraps into two lines right before the hash, so chances are the only thing they will see is the sentence. The full log message is: 2010-03-04 23:29:57 H=220-130-152-172.hinet-ip.hinet.net (ts100-e5.yotor.com.tw) [220.130.152.172] F= temporarily rejected RCPT : Sender reputation poor, retry sending later 282bf15886012068508968657d08940e,-76,1 > give them the first line, some give them the last line. ?Hence > this is why AOL sends exactly one line with an URL. Yes, I am a big fan of the way that AOL's system operates. Our postmaster page was originally inspired by AOL's postmaster pages, with one obvious exception that mine is woefully web 1.0 and AOL's looks beautiful :-). I was saddened to see them let go of so many people last month, and even more saddened that Analivia has moved on. But being the only one left has a completely demoralizing effect and to me it was only a matter of time before Analivia left too (having had the exact same scenario thrust upon me in the past, it only took a couple months before I found another job I was interested in and moved on). -- Regards... Todd I seek the truth...it is only persistence in self-delusion and ignorance that does harm. -- Marcus Aurealius From stuart at bmsi.com Sat Mar 6 12:38:00 2010 From: stuart at bmsi.com (Stuart D. Gathman) Date: Sat, 6 Mar 2010 12:38:00 -0500 (EST) Subject: [Pymilter] Too harsh? In-Reply-To: <48b1344b1003060808p4ef0450au2b1f06650d42c123@mail.gmail.com> References: <48b1344b1003021335h2a8d1f02m138a61c70c7e375@mail.gmail.com> <48b1344b1003060808p4ef0450au2b1f06650d42c123@mail.gmail.com> Message-ID: On Sat, 6 Mar 2010, Todd Lyons wrote: > to code up. The miltered version is not so easy. You probably will > chuckle when you read this next sentence: I had forgotten that a > milter cannot see headers that other milters have added. So any > attempt to create a reputation milter that does not also handle the > spam scoring will have to just log profusely and rely on some external > process to parse the logs and submit the feedback. I will probably With python milters, you just add additional modules, and they all share data in the connection object. (I need to create a formal plugin system.) -- Stuart D. Gathman Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154 "Confutatis maledictis, flammis acribus addictis" - background song for a Microsoft sponsored "Where do you want to go from here?" commercial. From dgc at uchicago.edu Sat Mar 6 17:35:52 2010 From: dgc at uchicago.edu (David Champion) Date: Sat, 6 Mar 2010 16:35:52 -0600 Subject: [Pymilter] Re: Too harsh? In-Reply-To: <48b1344b1003060808p4ef0450au2b1f06650d42c123@mail.gmail.com> References: <48b1344b1003021335h2a8d1f02m138a61c70c7e375@mail.gmail.com> <48b1344b1003060808p4ef0450au2b1f06650d42c123@mail.gmail.com> Message-ID: <20100306223552.GF2340@monkey.uchicago.edu> * On 06 Mar 2010, Todd Lyons wrote: > milter cannot see headers that other milters have added. So any > attempt to create a reputation milter that does not also handle the > spam scoring will have to just log profusely and rely on some external > process to parse the logs and submit the feedback. I will probably > use swatch for that, or maybe write a simple helper daemon that does > pretty much what swatch does. I haven't thought too much on that just > yet. When milter functions cannot be combined in a single milter, one solution is to use two layers of sendmail. One listens on *:25 and calls milter1, then passes to another sendmail running on, say, localhost:2501. Insert layers on localhost:2502, etc. as needed, one sendmail per milter. The final sendmail+milter instance delivers or performs final relay. If you pay Sendmail, Inc's Professional Services group a lot of money to help you stack milters, this (loosely and roughly) is what they will set up. > Yes, I am a big fan of the way that AOL's system operates. Our Ah, well, to each his own I guess. (My problem with AOL is their inane "feedback loop" strategy, which is more annoying than helpful to large peers, but which they firmly refuse to change.) -- -D. dgc at uchicago.edu NSIT University of Chicago From jafo at tummy.com Tue Mar 16 18:10:00 2010 From: jafo at tummy.com (Sean Reifschneider) Date: Tue, 16 Mar 2010 16:10:00 -0600 Subject: [Pymilter] Problem building on CentOS 5: 'unknown_callback' undeclared Message-ID: <20100316221000.GA25433@tummy.com> Trying to build pymilter on CentOS 5 with the stock sendmail-devel package, I get: miltermodule.c:947: error: 'unknown_callback' undeclared (first use in this function) If I apply the following patch to move the ifdef down afte rthe declarations, it builds: --- pymilter-0.9.3/miltermodule.c 2010-03-16 16:00:22.000000000 -0600 *************** *** 824,833 **** return generic_noarg_wrapper(ctx,abort_callback); } - #ifdef SMFIS_ALL_OPTS static PyObject *unknown_callback = NULL; static PyObject *data_callback = NULL; static PyObject *negotiate_callback = NULL; static int milter_wrap_unknown(SMFICTX *ctx, const char *cmd) { --- 824,833 ---- return generic_noarg_wrapper(ctx,abort_callback); } static PyObject *unknown_callback = NULL; static PyObject *data_callback = NULL; static PyObject *negotiate_callback = NULL; + #ifdef SMFIS_ALL_OPTS static int milter_wrap_unknown(SMFICTX *ctx, const char *cmd) { Thanks, Sean -- Do one thing every day that scares you. -- Mary Schmich Sean Reifschneider, Member of Technical Staff tummy.com, ltd. - Linux Consulting since 1995: Ask me about High Availability From stuart at bmsi.com Tue Mar 16 20:05:50 2010 From: stuart at bmsi.com (Stuart D. Gathman) Date: Tue, 16 Mar 2010 20:05:50 -0400 (EDT) Subject: [Pymilter] Problem building on CentOS 5: 'unknown_callback' undeclared In-Reply-To: <20100316221000.GA25433@tummy.com> References: <20100316221000.GA25433@tummy.com> Message-ID: On Tue, 16 Mar 2010, Sean Reifschneider wrote: > Trying to build pymilter on CentOS 5 with the stock sendmail-devel package, Good catch. I have built only with sendmail-8.14.3, which is how that sneaked in. Your patch is ok, but it lets callers think they have successfully set those callbacks, when in fact they are ignored. I will work on a patch to throw an exception from register(). Or is just ignoring them better? I'm not sure... -- Stuart D. Gathman Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154 "Confutatis maledictis, flammis acribus addictis" - background song for a Microsoft sponsored "Where do you want to go from here?" commercial. From jafo at tummy.com Wed Mar 17 09:17:52 2010 From: jafo at tummy.com (Sean Reifschneider) Date: Wed, 17 Mar 2010 07:17:52 -0600 Subject: [Pymilter] Problem building on CentOS 5: 'unknown_callback' undeclared In-Reply-To: References: <20100316221000.GA25433@tummy.com> Message-ID: <4BA0D680.6020904@tummy.com> On 03/16/2010 06:05 PM, Stuart D. Gathman wrote: > Your patch is ok, but it lets callers think they have successfully > set those callbacks, when in fact they are ignored. I will work > on a patch to throw an exception from register(). Or is just ignoring > them better? I'm not sure... I'm not really qualified to comment on whether doing an exception or ignoring them is better. The other thing I ran into was that getversion isn't available in 8.13, so I had to comment that out as well. That at least gets it building on CentOS 5 and able to be imported. Sean -- Sean Reifschneider, Member of Technical Staff tummy.com, ltd. - Linux Consulting since 1995: Ask me about High Availability -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 253 bytes Desc: OpenPGP digital signature URL: From stuart at bmsi.com Wed Mar 17 11:41:43 2010 From: stuart at bmsi.com (Stuart D. Gathman) Date: Wed, 17 Mar 2010 11:41:43 -0400 (EDT) Subject: [Pymilter] Problem building on CentOS 5: 'unknown_callback' undeclared In-Reply-To: <4BA0D680.6020904@tummy.com> References: <20100316221000.GA25433@tummy.com> <4BA0D680.6020904@tummy.com> Message-ID: On Wed, 17 Mar 2010, Sean Reifschneider wrote: > I'm not really qualified to comment on whether doing an exception or > ignoring them is better. The other thing I ran into was that getversion > isn't available in 8.13, so I had to comment that out as well. > > That at least gets it building on CentOS 5 and able to be imported. I am using CentOS 5 also, but I use sendmail-8.14.3. Consider getting the source RPM for sendmail-8.14.3 from pymilter on Sourceforge (you can use you own copy of sendmail source tarball and compare patches). You get all the latest milter stuff that way. Prebuilt binary is available also. -- Stuart D. Gathman Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154 "Confutatis maledictis, flammis acribus addictis" - background song for a Microsoft sponsored "Where do you want to go from here?" commercial. From stuart at bmsi.com Wed Mar 17 22:28:59 2010 From: stuart at bmsi.com (Stuart D. Gathman) Date: Wed, 17 Mar 2010 22:28:59 -0400 (EDT) Subject: [Pymilter] Re: Fwd: Milter SMFIC_HEADER failure (huh, due to success? :-) In-Reply-To: <0cd851d0-0f7b-4f96-b04e-d0714acbddcd@email.android.com> References: <0cd851d0-0f7b-4f96-b04e-d0714acbddcd@email.android.com> Message-ID: On Wed, 17 Mar 2010, Scott Kitterman wrote (forwarding from Sean): > Mar 16 12:50:11 mailhost postfix/cleanup[9490]: warning: milter > inet:127.0.0.1:2092: can't read SMFIC_HEADER reply packet header: Success > want this milter running on the submission port -- it is a NOOP for > recipients that are non-local. > > Any thoughts on this? There was a libmilter bug where negotiating for features like NOREPLY didn't work quite right with older sendmail. That might affect postfix as well. I'll try to find it when I get a chance, but this is a heads up. -- Stuart D. Gathman Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154 "Confutatis maledictis, flammis acribus addictis" - background song for a Microsoft sponsored "Where do you want to go from here?" commercial.