The Inbox Challenge

How many messages do you get a day? 20, 50, 100? Think again. On average, I get between 200 to 400+ e-mails/day. When spammers point their big guns at me (which happens occasionally) I can get as many as 600-800+ messages per day.

This screen shot (below) from my PerfectMail daily Activity Report e-mail for Nov 19, 2013, and tells me that 49 messages (about 20%) were wanted/delivered. The rest of the messages were blocked as spam.

I could go on about the productivity waste of having to manually screen almost 200 spam messages... but that's what a spam filter is for. Let's take a look at how well PerfectMail does it's job.

Separating the Wheat from the Chaff

Q. How do we find the 49 wanted messages out of all that clutter?

A. By applying a series of progressively more sophisticated real world tests designed to identify dangerous and unwanted messages.

Refused Mail

Malicious e-mail is any message that contains a payload (e.g.: attachment) that is designed to harm you or your computer. PerfectMail deals with these messages in 2 ways:

1. If an attachment contains a file that is any type of executable (.exe file, .bat file, .scr file, etc.) then the message is rejected outright.
2. If the attachment attempts to fake you out by making you think that the attachment is not what it is (e.g. a file named someImage.png.exe is an executable, not a graphics image file), it is rejected.
3. PerfectMail also extracts and scans all attachments with our embedded antivirus filter. Any hits from the AV filter and the message is rejected.

These tests that reject unwanted messages with very level of confidence. In my case, PerfectMail stopped 14 malicious messages.

Rejected Mail

Sometimes, you just don't want to talk to somebody. PerfectMail takes the same attitude against known spam sources. PerfectMail works with multiple Black List sources to safely reject e-mails from known spam senders. A Black List is a service that safely identifies spam sources by prior activity. These usually include:

1. Desktops compromised by a spam virus
2. Machines where the ISP or the domain owner has not authorized the machine to act as a mail source
3. IP address blocks that are owned by ISPs that don't care what their customers do (spam friendly networks)

PerfectMail can work with multiple sources for unwanted sender information. By default, it queries spamhaus.org but can be configured to use other sources. By safely eliminating messages from known spam sources, PerfectMail safely eliminated 128 messages from my inbasket today.

Outbound Mail

Why is Outbound e-mail in my daily activity report? Because this information can be very useful. With this information, I can:

1. Review my prior day's e-mail communications
2. Resend any e-mail from PerfectMail if my intended recipient loses it, mis-files it, etc.
3. Verify that only e-mail I intended to send actually went out (or said in another way, I can verify that I don't have a virus that turns my PC into a spam engine)

But clearly this doesn't affect my inbound e-mail... so let's move on.

Quarantined Mail

Mail that gets through initial checks (i.e.: does not contain a virus and does not originate from a known spam source) is subject to a wide range of additional checks. In a nutshell, these tests:

• Check the reputation of the sender (by mail server, e-mail address, domain, etc.)
• Check the message structure for reasonableness (message conforms to RFCs covering e-mail)
• Review the message content

While all of the above tests are important, the last item is especially so. Spammers usually craft their content to have you take some action (buy something, contact the Nigerian 419 scamer, visit an infected web site, steal your personal information, etc.).

To protect against these threats, PerfectMail includes the usual suite of content tests including:

• Black and White lists for words/phrases, domain names, IP addresses, etc
• Bayesian filters that look for the spamminess of a message

But, PerfectMail also has some very exciting (and fully automated) features to help correctly identify and block spam. Here are just some of it's capabilities:

• Automated HoneyPots. A Honeypot is a baited trap. In our case, we tell a small percent of spam senders that an invalid recipient e-mail address is actually valid. Spammers then send e-mails to that non-valid address and CC other valid users. The use of the non-valid address on a recipient list allows us to safely block the message for everyone
• Website Reputation Checks. PerfectMail extracts URLs in an e-mail (possibly following many redirects). If we see that the IP address of web server is on a reputable black list, we can safely reject the message
• Website Content Checks. PerfectMail can pull down the content of any target web pages (referenced in an e-mail). This content is checked to see if the site is malicious, promotes bogus pharmaceuticals, etc.
• Character Set Checks. HTML e-mails declare the character sets (alphabets) used in the message. There are many character sets including Western, Cyrillic (Slavic), Chinese, Japanese, Turkish, etc. By watching outbound mail flows, PerfectMail learns what character sets you use - and don't use. If a spammer sends you a message using the Turkish character set (for example) and you have never, ever sent anything in Turkish, then the message is very likely to be blocked

PerfectMail placed 39 messages into our quarantine due to content related concerns.

The Result

Through a combination of tests, from basic to advanced, PerfectMail thoroughly examines all aspects of an inbound message. Messages are scored when we see questionable behavior (attempts at abuse or not following e-mail best practices). If a message score exceeds a preset maximum, the message is saved in the quarantine and not delivered to the recipient.

To protect against false positives (incorrectly blocking wanted e-mail), PerfectMail keeps a copy of all content blocked messages in a local message archive. It sends all protected users daily activity reports with a list of all blocked messages. From these reports, users can review and release wanted messages from the quarantine.

How effective is the quarantine? My experience is typical of most PerfectMail users. In a normal week, I release no more than 2-3 messages from the quarantine (out of 1,750+ (7 days x 250 messages/day)). This yields an accuracy rate of about 99.8%

My inbox is clean and uncluttered with spam. Because of PerfectMail's very high accuracy rate, I spend my precious time on valid e-mails and not rummaging through the quarantine.

-- Larry Karnis