[Remops] EMP Filters and Me

[Bananasplit Admin]

On Sat, Mar 14, 2009 at 10:36:12AM -0400, drsnoid wrote:
> 
> Well, there is something amiss then.  If I compare those messages which
> according to the mail2news log have either been accepted and posted or
> reported as existing duplicates with what actually appears in
> alt.testing.testing, there are very many missing.  This is evident on any
> news server I read from, and interestingly news.mixmin.net has the *fewest*
> - - one would expect it to show the greatest according to the mail2news log.

Indeed, and if some are going missing I'd like to understand why.  Do
you have any sample Message-ID's of ones missing on Mixmin but posted
elsewhere?
 
> Ah, I wasn't aware that posting-host was a part of the equation.  I've also
> wondered if age were one of an EMP filter's criteria (does "substantially
> identical" expire after n days?)

No.  There are primarily three types of EMP filter:-
PHL	-	NNTP-Posting-Host / Lines
MD5	-	Message Payload
FSL	-	From / Subject / Lines
All of these are created by MD5 hashing the criteria.  The hash is then
stored in a list along with a counter.  Each time a hash collision
occurs the associated counter is incremented until a threshold is
crossed and further messages are rejected.  Over time the counters
decrement until they reach zero and the hash is deleted.

> Do you think my choice of alt.testing.testing rather than say,
> alt.test could have any bearing?  Didn't use to, but clearly something
> has changed.

I think alt.testing.testing is an excellent choice, it's not used for
much else but is help on most news servers.

> What would be the best course of action for nefarious me to take in
> circumventing any EMP filter?  In other words, what exactly quantifies
> "substantially identical"?

The three filters I described above are the main ones but none of them
should be rejecting messages to test groups.  If they were, you would
see the reject messages in the mail2news log.

> Finally, I think that at least two exit remailers are performing some sort
> of EMP filtering on messages as they emerge *before* they get to the
> mail2news - cripto and paranoia.  I might be wrong but their behavior seems
> to indicate it.

Many remailers used to run nilsimsa filtering.  Nilsimsa creates article
hashes that are substantively the same for similar messages.  This was a
defence against the old days of heavy remailer flooding.  Don't know if
these two did (or still do) it.

Sorry for not responding sooner.  I had to visit Twickenham yesterday
and watch England vs. France.  Tough job but somebody has to do it.  :)

-- 
()  ascii ribbon campaign - against html e-mail 
/\  www.asciiribbon.org   - against proprietary attachments
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: Digital signature
URL: <http://lists.mixmin.net/pipermail/remops/attachments/20090316/bc5153b6/attachment.pgp>