Greg's Head

I’ve started reading spam. It’s not glamorous but there sure is a lot of it to read. When you start reading spam you quickly see some pattern.

1. 98.9% of spam comes from people you don’t know. About .005% is from a friend who forwards me jokes or pyramid schemes. The remaining 1% is from websites that I’m registered too that send holiday emails.
2. Spam communicates it’s message in two core ways:
   • Links to websites
   • Images within the text that direct users to call, email or open a web-page
   • Some messages have no link or actual content and seems to be trying to confuse spam blocking software so that future messages can get through.
3. The text of a message tends to be semi-random meaning each message is semi-unique. Sometimes words have random characters, sometimes there is non-sense words and other times it seems to be random text copied from a random website.

I realized that modern spam filters have some serious problems:

- Bayesian filters will get tricked by random messages and content taken from the web
- Community filtering can’t flag all messages if the content is semi-random
- Image filters can’t recognize the text in an image if the image is random and the text in the image is sometimes random.
- Dynamic blacklisting can reduce spam but not eliminate it.
How about an alternative?

I’ll try to avoid the term ‘filter’ because this word implies that messages can get through the filter by tailoring content. No matter how smart the filter gets it’s possible to craft messages that will pass through. Any system that relies on ’spam filters’ will have both false positives and false negatives.

Instead of filters let’s start with a basic trust system.

What’s a trust system? Ebay uses a good example of a trust system. Users establish trust and reputation over time. Anyone can give feedback on the trust relationship. The longer you have an Ebay account the more your trust grows over time. The same should be true for email.

There are many ways to build such a system but the easiest way to explain it is by comparing it to the Google page rank technique. Assume that every email I send is like an outbound link. Every email I receive is an inbound link. With your own email data you could compute the email rank of any individual. With an aggregated data set you could compute the email rank across a very wide range of people who use email.

Each email you receive allows you to give basic rating of positive and negative. Really simple.
New Users and Mailing Lists
When a new user enters the system they would have an email rank of zero. Their initiation into email would be to build a trust relationship with someone who has a higher rank. As the email is used the persons reputation grows and the validity of their messages increases.

Mailing lists that people subscribe to would have to build the same type of trust. As these mailing lists are used and as more people subscribe/opt-in the trust of the list grows. At Raizlabs we have a mailing list product and I’m always nervous that software designed to be used as a useful tool for news and opt-in mail could be used by someone to send out spam.

Why is it that I know more about the trustworthiness of random people on Ebay then I do of the trustworthiness of email in my own inbox?

For more ideas on email check this out.

Over the last 10 years a basic etiquette to sending email that has evolved. These rules are considered by some to be common wisdom, unfortunately this wisdom isn’t as common as most would like. Following a basic etiquette is fairly simple and it makes reading and receiving lots of email easier.

• Add a subject line to every message.
  A subject in an email is like the envelope for a letter. The subject should be short concise and to the point. Adding a subject line to each email makes it easier to read, find and filter. Adding a subject also makes it easy to quickly find a particular email from hundreds or thousands.
• ALL CAPS
  Typing something in all caps is often synonymous with shouting in an email. All caps messages are harder to read. To turn off caps use the caps lock key on your keyboard, it’s located next to the letter “A.”
• Reply All
  If you are emailing a small group of people (2-5) it’s ok to reply to everyone but you should avoid using “reply all” with larger groups or email distribution lists that may contain many people. If your reply is directed at only a couple people don’t send your message to the entire list.
• Replying to a message
  When you reply to a message it’s customary to keep the original message bellow the reply. This isn’t necessary but is often helpful if you re-read your mail at a later point in time.
• Forwarding email
  Do not forward chain letters, jokes, internet myths or other random mail. The only exception is if the person has explicitly asked for this. If people really want this type of email there are plenty of joke-a-day lists on the internet.
• To me or to CC that is the question
  If you need the person to read your email place their name on the To line. If you don’t need the person to read the email but thought they may want to know place them on the CC line. Do not assume that every CC mail will get read. People who are busy may filter their CC messages into a separate folder or not read them at all.
• Large Attachments
  Avoid attaching large files (2 megs or larger) to an email. If possible use a link to the file or image. Large files can often get rejected by servers, if the recipient is on a slow connection this can bring their email to a halt. If you need to send a large file consider storing it on a server and then just send a link to the file.

  If you find these rules useful, print out a copy, tell a friend, post a link to your site, mail it in a letter, just don’t send an email about it.

The initial version of email has a lot of problems that can not be addressed by making minor modification to the way that email is trasnsmitted. A new architecture needs to be developed to begin to replace our current email infrastructure.

Problems
- No guarantee or acknowledgement of delivery.
- No accountability for who sent the message (spam)
- No guarantee on latency or delivery
- No general security (email is sent as plain text)
- No standard messages for bounced, inactive accounts or full accounts

Our current system:

Greg -> Server1@host.com  <- net ->  Server2@host2.com <- Kathy

Today Greg sends a message addressed to Kathy@host2.com. Greg’s mail server releases the message onto the internet. The message is sent back and forth between servers untill it reaches host2 or untill the servers give up. Greg has no way of knowing if the message was delivered. Kathy has no way of knowing if the message recieved is legitimate.

Proposed solution :
Greg types his message and sends it to Host1. The host creates just an envelope consisting of an encrypted key for the message, Greg’s name and the address of Host1. This envelope is sent across the internet in a similar fashion to our current email system. If Host2 recieves the envelope it can connect back to Host1 and retrieve the message securly using the encrypted key provided in the envelope. If Host2 never recieves the envelope host1 can automatically resend the envelope.

Send just the envelope wrapper using Email 1.0

Greg -> Server1@host1.com  <- net ->  Server2@host2.com <- Kathy

Once Kathy’s server recieves the envelope it connects securly back to Greg’s host and downloads the body of the email.

Server1@host1.com  <- net  <- Kathy

This is better then the current system for several reasons

1. This makes it much harder to spam. Currently after a message is sent the sender can disapear on the internet. The new system requires that the mail server holds outgoing mail and waits for it to be retrieved. This provides a greater level of accountability. Additionally it means that each message is recieved from the correct host. Since the reciever connects to the sender computer to retrieve the message it is no longer possible to forge email addresses unless you have control of the sending host computer.
2. Once a message is sent it can be canceled or corrected before it is retrieved. Everyone who has used email has hit send and wished they could fix, correct or cancel their message. This new architecture would make it possible. When you hit send you are now just sending an empty envelope the message is only retrieved when the server wants to download it.
3. The new approach allows you to tell when a message has been successfully delivered. Since each envelope is unique and has a unique key you could easily tell when the message was retrieved. This doesn’t prove that the message was ‘read’ but at least you know it didn’t get lost along the way.
4. Initially the system could be made to be backward compatible with current email networks. Initially the envelope could also contain the body of the message as well as the secure key. Email software that is aware of the key could download the message securly to help authenticate the sender. Email software that is not aware of the security key would just see the message. There would be incentive to upgrade your email client since it helps ensure that messages sent are delivered. Recieving less spam would also be an insentive to upgrade. In addition many of the clients could be updated transparently by simply updating the backend server and still delivering mail to the client applications using traditional pop3 and imapi. Another way to grow adoption and still have backward compatibility is to format the message in a way that brings users to a secure location:

   You have recieved a secure message from: ‘Greg’ who is using Email2 for secure communication. It seems your server does not support email 2. You can read this message by installing email2 software or by clicking this link: https://host1.com/key/23232082959372

5. Email communication would be more secure. Although there is no perfect way to secure email commuunications the new system would be more secure then the curent system.

Having a framework for developing a new email technology would allow for other advanced that are desperatly needed.

• Standard subscribe / unsubscribe headers
• Standard bounce messages
• Standard threading and referenceing (rather then including messages in each mail)

I think a revision to the way Email is sent and recieved is long overdue. Even if this isn’t the exact solution we really need to rethink the way that Email should work.

Email notification is a sweet seducer of your time. You’ve got mail! It’s exciting, someone cares enough to write something to you. The sound of the new message, the pop-up notification begs you to stop what you’re doing and read it!

I think that email is to some degree addictive. Each message is a bit of mental sugar, a little nicotine of digital satisfaction. I have found myself checking email all the time. This could even be made worse by programs that check automatically for you.  I’m guilty of writing one. Doh!

Want to really increase your productivity? It’s simple. Set your email checkers, outlook, blackberries, etc. To check for mail only once every 1/2 hour or even once every hour if you’re brave. Less checking means less distractions. Less distractions means longer stretches with more productivity.

• Less checking cuts down on quick back and forth messages. Email isn’t a great tool for an “instant messenger” conversations. Use a phone or meet in person to discuss. Horror!
• You may find that smaller issues get resolved on their own.
• If several people have replied before you it gives you the opportunity to have the last word. This also means you are more likely to be in a role where you can break a tie in the direction you want.

The inital version of email has a lot of problems that can not be addressed by making modification to the way that email is trasnsmitted. A new architecture needs to be developed to replace our current email infrastructure.

Problems
- No garentee or acknowledgement of delivery.
- No accountability for who sent the message (spam)
- No garuntee on latency or feedback of delay
- No general security (email is sent as plain text)
- No standard messages for bounced or inactive accounts

Today:
Greg ->MailServer1@host.com <– net –> MailServer2@host2.com <—- Kathy

Today Greg sends a message addressed to Kathy@host2.com. Greg’s mail server releases the message onto the internet. The message is sent back and forth between servers untill it reaches host2 or untill the servers give up. Host1 has no way of knowing if the message was delivered.

Proposed solution

Greg types his message and sends it to Host1. The host creates an envelope consisting of an encrypted key for the message, Greg’s name and the address of Host1. This envelope is sent across the internet in a similar fashion to our current email system. If Host2 recieves the envelope it can connect back to Host1 and retrieve the message securly using the encrypted key provided in the envelope. If Host2 never recieves the envelope host1 can automatically resend he envelope.

This is better then the current system for several reasons

1- This makes it much harder to spam. Currently after a message is sent the sender can disapear on the internet. The new system requires that the mail server holds outgoing mail and waits for it to be retrieved. This provides a greater level of accountability. Additionally it means that each message is recieved from the correct host. Since the reciever connects to the sender omputer to retrieve the message it is no longer possible to forge email addresses.

2. Once a message is sent it can be canceled or corrected before it is retrieved. Everyone who has used email for more then one year has hit send and wished they could fix, correct or cancel their message. This new architecture would make it possible.

3. The new approach allows you to tell when a message has been successfully delivered. Recipt of the encrypted key shows that the message was recived.

4. It may be possible to make this email servers that supported the new transport but also have backward compatibility with pop and SMTP email clients. New client software could give additional features such as canceling a sent message or checking the status on any sent item.

5. Email communication would be more secure. This system would help prevent email forgery. It would also prevent a ‘man in the middle’ scenarios.

6. Because this new technology builds on existing systems it can be backward compatible with older clients. For example the envelope that is sent by Host1 can contain user readable instructions for reading the message:

• You have recieved a message from: ‘Greg’ who is using Email2 for secure
  communication. It seems your server does not support email2. You can read this message by installing server software or by clicking this link:

  http://mailserver1.host.com/message/232959372

Proposed steps:

Greg ->MailServer1@host.com
Greg sends his message to his own mail server addressed to another recipient

MailServer1 <— net —> MailServer 2
The mail server sends an envelope to MailSerever 2.

If Kathy @ MailServer2 would like to read the message a secure connection is made directly from MailServer2 and MailServer1 to retrieve the content of the message.

Points to consider

1. The burdon on the sending mail server is proportional to it’s volume of email. So if a spammer tries to send a million messages he needs to have the website to support a million people reading his message. If his DSL company cuts him off any unretrieved messages would get discarded.
2. Attachments, and other large files would not ‘clog’ your inbox because they would only be retrieved as needed when the message was being retrieved.
3. An email message can become a dynamic, possibly editable document. Once you hit send the document doesn’t have to get stale.
4. Mailing lists become easy to manage because you can determine what email addresses are no longer active.