Tuesday, October 20, 2009
You may have noticed some emails from your EditMe wiki site didn't come through last week. If you follow EditMe on Twitter @EditYourWeb you probably know why:
If you're not familiar with Spamhaus, it is an international non-profit group made up largely of volunteers that provides realtime anti-spam protection for Internet networks. Spamhaus works with law enforcement and network operators to identify, pursue and eradicate sources of spam. They provide this service via a globally distributed database that network administrators can plug into (for free) to determine from which email servers they should block (or suspect as spam) incoming email messages.
All email originates from an email server somewhere on the Internet. If Spamhaus detects a mail server sending unsolicited spam, they add it to their database of blocked servers. Network administrators using the Spamhaus database will then either block or tag incoming mail from these servers as spam. How this information is used is entirely up to the individual network administrator (e.g. the IT guy at your company). Spamhaus only points the finger. But it's a very powerful finger, protecting nearly 1.5 billion inboxes.
EditMe sends email on behalf of our customers' wiki sites. While perhaps not a mission critical aspect of the EditMe service, these notification emails are an important part of our customer's workflow.
So when I caught wind that customers weren't getting emails, Spamhaus was the first place I checked. And sure enough, the IP address of EditMe's main mail server had been added to the blocked list.
When I contacted Spamhaus about this, asking that the address for EditMe's mail server be removed from the blocked list, I received this unnerving response:
Now, as you may know, EditMe hosts its wikis in Amazon's EC2 cloud computing service. So I searched Amazon's support forums to see if I was the only one who'd noticed this problem.
Sure enough, I wasn't. A thread at the EC2 forums had already been created. I added my two cents, asking Amazon to contact Spamhaus and get this resolved. After all, this block was stopping mail from ALL of Amazon's EC2 customers to the almost 1.5 billion inboxes protected by Spamhaus block lists.
Thankfully, Amazon did the right thing by their customers and responded (5 hours after it was first reported) that they were in contact with Spamhaus. Seven hours later, they updated us with a note that the block had been reduced in severity, but still persisted.
Proof that AWS support staff are working around the clock, an update at midnight informed us that the block had been removed and that AWS would work with Spamhaus in an effort to prevent such a block in the future.
If you take the time to read through the lengthy forum topic tracking this issue at the AWS support forum, you'll find an interesting debate that ranges from suggestions for AWS to prevent spam, to vehement attacks on Spamhaus for so wrecklessly blocking such a vast pool of business servers.
At the end of the day, Spamhaus says they tried to contact Amazon before the block was put into place. If you've ever tried to contact anyone at AWS, you know it's not easily done - they essentially don't provide contacts unless you're a customer paying a support premium. Amazon provides a contact form to report abuse, which is probably what Spamhaus tried. I reported this issue via that form and got no response whatsoever.
When Spamhaus got no response, they slapped the cuffs on and waited. Sure enough, that got Amazon's attention. Hopefully, in the future, Amazon will respond a bit more quickly to such queries rather than waiting for it to get to a critical stage. They surely will lose some customers over this.
Amazon has posted to their support forums that, because of this issue, they have instantiated a new policy to limit outbound email of new customers. In order to send large amounts of mail from an EC2 instance, customers must fill out a form to justify their use case. Great news!
