[Pymilter] general gossip questions

[Todd Lyons]

On Fri, Feb 26, 2010 at 12:21 PM, Stuart D. Gathman <stuart at bmsi.com> wrote:
>
>> I was able to telnet multiple times to the daemon in my testing and
>> never had problems with concurrency, so I'm not sure that I even
>> comprehend what the REUSE_ADDR issue actually is.
>
> Telnet to daemon.  Restart daemon with session still active (something
> that will almost always be the case in production).
>
> Daemon will shutdown for restart, but won't start again for 5 mins.
> Error in log is "socket in use".  The SO_REUSE_ADDR socket option is
> supposed to let you immediately reuse the socket without trying to
> shutdown active connections.
>
> Somehow, the allow_reuse_address flag doesn't actually result in the
> socket option getting set.  I have spent a little bit of time debugging the
> TCPServer python code, and can't see where it goes wrong.

I have some more detail on this, but still no solution yet.  Maybe
this feedback will spark something in your thought process, or maybe
it will just be recounting what you've already seen on your own
system.

Recall that I setup a two node system.  The mail servers are named
ivwm51 and ivwm52, each with pygossip running and configured as peers
of each other. *  The exim on ivwm51 is configured to talk to pygossip
on ivwm51, and exim on ivwm52 is configured to talk to pygossip on
ivwm52.  I see messages in both logs reflecting peer responses and
reputation.  No problem.  Traffic is flowing at a couple messages per
second on each machine, not an overly heavy load.

Now I restart pygossip on ivwm51.  It gets the error you speak of.  At
this point, exim on ivwm51 can query the pygossip on ivwm51, pygossip
on ivwm52 can query the peer on ivwm52, but the pygossip on ivwm52
cannot query the peer on ivwm51.  Ok, control time:  I stop and start
pygossip on ivwm52.  Now ivwm52 can get peer information from ivwm51
again, but ivwm51 can no longer get peer information from ivwm52.  The
only way to fix it is to stop both of them and start them both
simultaneously.

I have a theory about this.  The big red flag for me is that if peer 1
is restarted, peer 2 can no longer get info from it.  Restarting both
simultaneously works properly, and the act of opening port 11900
before any client calls to peers are initiated seems to be a
contributing factor whether it will succeed or not.

I hope this description makes some sense.

* Side question:  For the peer list, you don't include the pygossip
running at localhost as one of the peers, do you?  On ivwm51 I put
ivwm52 as a peer, and on ivwm52 I put ivwm51 as a peer.

* Second side question:  Do I increment the TTL for each peer that I
add to the list?  If I added a third pygossip server ivwm53, would I
need to increment the TTL?  It _seems_ to me like I would not have to
(i.e. it seems like a hop count), but I'm not 100% sure I'm reading
the code accurately.

>> I plan to put pygossip on two servers, configure them as peers, and
>> set my TTL to 2.  Then I've got 8 servers that I will point at those

I have the TTL set to 1 not 2.

> Note that Gossip is compatible with "AOL style" user feedback as well
> (where user complaints about an email form a reputation that will eventually
> get a sender kicked off AOL.  AOL tracks IPs rather than domains).

I already extract IP information for an internal RBL from ARF
formatted feedback emails, so I don't think it would be too difficult
to add another subroutine to create a UMIS and then submit spam
feedback for it during the ARF processing.  That's a great idea,
thanks!

Now I'm working on the milter (sorry, but I prefer perl :-), will let
you know how it works out.

-- 
Regards...      Todd
I seek the truth...it is only persistence in self-delusion and
ignorance that does harm.  -- Marcus Aurealius