Fighting a bazillion ways to spell v1*A*g*ra...

Well... I've been thinking about spam further the past few days... In particular, how there are so many ways to spell various spam words.

It reminded me of a project I was working on to do data clean-up of some address information for a database of addresses that spanned the country... Customer Service Reps (CSRs) were essentially creating multiple address records by entering data inconsistently, and that this data had been collected over several years with little validation, but was a key piece of data for searching the system...

We created multiple scripts to rate the likelihood that 2 addresses were actually the same, and to merge these addresses... In our system, they were case sensitive, so that :
123 Main St.
and
123 MAIN ST.

Ended up being the same record, and could be merged... We added further logic to consistently map Street to St., Drive to Dr, etc etc.. (there were about 100 street suffixes...)
So we would also recommend merging
123 Main Street

Now some cities also had East / West...
123 Main St. East and 123 Main St. West were probably not the same... But 123 Main St. and 123 Main St. East could be the same - if postal code / zip codes matched also... Consider that the CSR might type W, West, West, or for data in Quebec (Oueste, or even just O) - so we needed mapping rules there too...

Then there were the typos... Young St. vs. Younge St., vs. Youngge St... We tweaked the script to modify the score if, we removing vowels or duplicate letters (two g's in a row) ... Matches increased the likelyhood that much more...

This same kind of logic could be applied to spam... Map \/ to V, map all of the accented characters to the values that might appear as (we did something similar for accented characters in our Quebec addresses as well)... 1 become i, strip spaces, remove asteriks, etc, etc...

Pretty soon you end up with "buyviagranow" and you can then pick out the evil phrases and you're laughing...

Our technique for finding matches was pretty strong - unfortunately it turns out that the CSRs were taking advantage of the way addresses got stored seperately to work through a system / design flaw... Weeks of migration work had to be backed out... But the mapping exercise was still an intersting / challenging examination at handling / validating / mapping relatively complex bad data...