[Pymilter] general gossip questions

[Todd Lyons]

On Mon, Feb 22, 2010 at 9:00 PM, Stuart D. Gathman <stuart at bmsi.com> wrote:
>
>> don't have a need for, so now I'll see about getting pygossip up and
>> running on a test mail system.
> Beware the unsolved problem of REUSE_ADDR not working for TCPServer.

I googled to try to understand a little bit about this.  What prevents
you from just adding:
SocketServer.TCPServer.allow_reuse_address = True
right after the import SocketServer?  Or does that not fix the issue?
I was able to telnet multiple times to the daemon in my testing and
never had problems with concurrency, so I'm not sure that I even
comprehend what the REUSE_ADDR issue actually is.

> Correct.  Easy, and sufficient for my servers.  My server with
> heaviest usage (100000 msgs/day) uses 630M for the shelve database
> with no problem.  I think there is a way to plug in another database.

I plan to put pygossip on two servers, configure them as peers, and
set my TTL to 2.  Then I've got 8 servers that I will point at those
two instances.  I figured I'd have 4 point to one pygossip server and
the other 4 point to the other pygossip server.  Since the TTL is 2,
they should talk to each other and trade reputation info back and
forth.  This should also result in the data being evenly split between
the two machines.

Or am I misunderstanding how the peer system works and is all data is
stored on both nodes?  I apologize for the lack of understanding this
question reveals.

> The 'R' command resets the reputation for a domain:qual.  The tc.py
> script (inexplicably not installed by package - should at least go in
> /usr/lib/pymilter) provides a simple command line interface to query, feedback,
> and reset reputations.

That's awesome, in all my reading I had never seen the R command.  I
also added tc.py into the rpm I build.

Part of the reason I think I didn't know about the R command is that
the pygossip webpage download link goes directly to pygossip-0.3.  It
wasn't until after I sent my first email to the list that I realized
there was a pygossip-0.4.  Since I had been looking at gossip and
pygossip off and on during the previous week, you would think I would
have stumbled across that R command, but I never did (because I was
reading 0.3 release code).

> The UMIS identifies a message - as opposed to a domain qual.  The
> UMIS is stored in an email header, and can be used to update how that
> message "votes" for the reputation of the domain.  The UMIS is used
> so that each message gets at most one vote.  pygossip doesn't currently worry
> too much about losing votes.  :-)

That seems pretty clear, thanks.

> Note that IP has been generalized to qualifier.  In the milter package,
> domains that pass SPF have the SPF qualifier.  Softfail gets the
> SOFTFAIL qualifier, and so on for NEUTRAL, FAIL, PERMERROR, GUESS (gets
> a best guess pass).  (Yes, some braindead domains routinely send out mail with
> SPF fail, and need a special policy to accept their braindamage - but the
> reputation system will still block them if enough spam comes in with their
> domain.)  IP is used when SPF result is NONE.
>
> Thus, the NEUTRAL reputation of a domain may have a different (and likely
> much worse) reputation than the SPF reputation.

Since I'm going to be implementing my own feedback mechanism in exim
and writing my own milter for sendmail, I have a blank slate WRT how I
want to treat all of these variables.  I will definitely be studying
your code and drawing from its concepts and design.  At this point the
variables I want to use for generating a feedback score are:
  spamassassin score as the base
  spf result
  dkim result
  helo name tests

There could easily be more, this list is off the top of my head.

Thanks for the feedback Stuart, much appreciated.

-- 
Regards...      Todd
I seek the truth...it is only persistence in self-delusion and
ignorance that does harm.  -- Marcus Aurealius