Spam Nation (5 page)

On the afternoon of November 11, I sent several months’ worth of data detailing McColo’s offenses to the company’s two ISP partners that connected it to the larger Internet: Global Crossing and Hurricane Electric, both of which had headquarters in the United States. The information was arranged in a map that showed how the servers used to control all of the top five most active spam botnets—Internet-connected programs responsible for sending most of the world’s junk email—were parked at just a handful of servers in McColo’s Northern California hosting facility. I had a hunch that, once presented with the record of malicious activity there, McColo’s Internet partners would sever business ties with the hosting provider and effectively cripple it.

Hours later, I heard from a source who monitored global spam activity daily, and who knew I was working on a piece about McColo.

“Krebs, what did you
do
?” the source asked with a praising laugh. “I’m hardly seeing any more spam, and it looks like McColo has been unplugged from the Internet!”

I don’t recall saying thank you or good-bye—I only remember swearing loudly and slamming the receiver down to quickly dial several other sources on my mobile phone. All of them confirmed the same findings: McColo was gone, and none of its Internet address space was reachable from anywhere on the World Wide Web. Mission accomplished—for the time being.

A call to Benny Ng, Hurricane Electric’s director of marketing, revealed the reason. The ISP had severed ties with McColo that afternoon.

“We looked into it a bit, saw the size and scope of the problem you were reporting, and said ‘Holy cow!’” Ng said. “Within the hour we had terminated all of our connections to them.”

Within a few minutes of confirming the takedown, I wrote and published a blog post about the McColo outage—which quickly became one of the biggest cybercrime stories, in terms of immediate global impact, up until that date—and then began working on a longer story about the incident that was intended for publication on the
Washington
Post
’s site and possibly in the dead-tree edition (as the print version was affectionately known among us dot-com reporters) the following day.

I worked from my home office that evening and well into the morning, toiling over the follow-up piece until eventually falling asleep in my pajamas at the computer keyboard as I finished the story around dawn.

The piece was edited and published on washingtonpost.com later that morning, and for a brief time the story was featured “above the fold” as one of the most popular on the site that day. That is, until a lawyer for washingtonpost.com found it and went positively ballistic. Apparently, nobody had asked the lawyers for their input, and now the attorneys were clamoring for the story to be unpublished from the website until facts could be triple checked and certain language about alleged illegal activities at McColo could be toned down.

Editors at the
Washington
Post
and other major publications typically request that a pending story be “lawyered” when it contains statements of fact or allegations that could lead to legal trouble down the road, particularly from the parties named in the story who might wish to pursue libel charges. One washingtonpost.com lawyer was extremely uncomfortable with any language that even hinted at illegal activity on the part of McColo’s owners, who had repeatedly ignored requests for comment. (To give a sense of how shady the dealings at McColo were, the sole points of contact listed on its website were anonymous instant messenger accounts.) After all, there was no evidence that anyone associated with McColo had been charged with any crime, so why were we alleging it?

(An important note: The story that ran that morning was full of links to supporting evidence of illegal goings-on at McColo, as gathered by countless security experts in the industry. Unfortunately, the washingtonpost.com lawyer who objected to it being published initially viewed the piece on her mobile phone, which had stripped out all of the hyperlinks that readers could use to view voluminous third-party reports and evidence of said criminal activity. To the attorney, the story appeared to be hurling all kinds of baseless and potentially libelous accusations at McColo, whose business at this point seemed all but ruined.)

The attorney demanded that the McColo story be pulled from the washingtonpost.com website, and after a brief period of defiance, the website news desk acquiesced without asking me whether the story was accurate or what supporting evidence I had to back up my reporting. The piece was simply yanked off the site, with no explanation to the tens of thousands of readers who found dead links and were eventually redirected back to my original blog post about the takedown. My inbox quickly filled with emails from mystified readers wondering where the story had gone.

For nearly five excruciating hours, the follow-up to one of the most important cybercrime stories to date remained in editorial and legal limbo, as the lawyers hashed over the piece line by line, changing or deleting potentially objectionable bits and pieces.

The piece was eventually republished later that evening, albeit in a shorter and much redacted form. But from that day forward, any story of mine that contained even a whiff of information about alleged online criminal activity had to be forwarded to at least one senior editor at washingtonpost.com and often run through a gamut of lawyers. Since I considered my beat to be cybercrime, this usually happened several times a week.

After the McColo fiasco, investigative stories that took weeks and sometimes months to produce could sit just as long in the inboxes of higher-ups whose approval I had to get before the stories could be
published. In some cases, subsequent stories were placed on indefinite hold by washingtonpost.com editors, the lawyers, or both.

One of those pieces was an investigative story I’d spent six months reporting and writing, about a pattern of cybercrime activity that traced back to Vrublevsky’s ChronoPay. At the time, the fastest growing and most lucrative cybercrime scheme worldwide was the spread of fake antivirus software. Also known as “scareware,” fake AV uses misleading pop-up alerts and other ruses to frighten unsuspecting Internet users into purchasing worthless security software. Adding insult to injury, the bogus security programs often are bundled with malware that turns host machines into spam zombies.

Security experts who had been closely tracking the scareware scourge told me they’d found that ChronoPay was nearly always responsible for processing the credit card payments for scareware scams, and that the company’s founder—Russian Pavel Vrublevsky—appeared to be heavily and personally involved in engineering and profiting from these schemes.

I knew very little about Vrublevsky until late 2008, when a Russian source (who will remain anonymous) urged me to look up ChronoPay’s incorporation records in the Netherlands, where ChronoPay was founded. Those records showed that ChronoPay was created in 2003 as a fifty-fifty partnership between Vrublevsky and Igor Gusev. The same sources that led me to the incorporation data said that in 2005, the two men parted ways. Gusev would go off in 2006 to found the GlavMed-SpamIt rogue online pharmacy partnership. Not to be outdone, a year later Vrublevsky would cofound Rx-Promotion, a competing rogue Internet pharmacy.

I had no clue about Vrublevsky’s ties to Rx-Promotion at the time, or even who Igor Gusev was. What I did know was that ChronoPay had very recently been associated with the Conficker worm, a computer contagion that remains one of the most virulent and heavily scrutinized strains of malware ever unleashed. An early version of the worm
instructed millions of infected computers to download a rogue antivirus program from Trafficconverter.biz, an online business that made tens of millions of dollars by paying scammers to foist fake antivirus software on PC users. And ChronoPay was the company responsible for processing payments for TrafficConverter.

In March 2009, I turned in the first version of an exposé on ChronoPay’s pivotal and lucrative role in the spread of fake antivirus software. The piece also presented evidence indicating that Vrublevsky was the founder, owner, and creator of Crutop.nu, the shadowy online forum that catered to the spammers and scammers who had attended McColo’s funeral.

The story cited published research from several esteemed security experts about ChronoPay’s history. Nevertheless, it was held in editorial limbo for months, punted from one washingtonpost.com senior editor to another. The editors were convinced ChronoPay would sue the
Washington
Post
, which was understandable. In our phone interview, Vrublevsky had promised his company would do just that if we ran the story.

The same dithering delayed another big scoop related to the ChronoPay piece. When McColo went dark, much of the illegal activity that had made its home there quickly shifted to another Northern California hosting provider, Triple Fiber Networks, or 3FN as it was known in the underground. The same spam botnet controllers that had called McColo their home for years had begun using 3FN after McColo’s demise. A review of postings at the online forum Spamdot—a closely guarded virtual den of thieves where most of the most successful Russian spammers gathered at the time—showed that 3FN’s owners actively picked up McColo’s stranded customers when the company’s operations were shuttered in November 2008.

At the time, 3FN also was the Internet’s largest host of sites that pushed fake antivirus software. The 3FN website was eerily similar to McColo’s. Again, the only way to contact the company’s owners
was through ICQ (“I seek you”), an instant messaging protocol that for years was the de facto communications medium for many Russian hackers.

I pressed for these stories to come to light. Having exposed the malicious activity that eventually knocked RBN, Atrivo, and McColo offline, I believed that the
Post
had an obligation to its readers—and to the wider world—to keep the spotlight trained on those Internet providers that offered safe haven to a huge swath of the cybercrime community. Bad press on these companies from major media would force more law-enforcement agencies into taking action against them and thus reducing the threat they posed both to Americans and people all over the globe. But my editors were hardly anxious for a repeat of the McColo story, even though it hadn’t resulted in any lawsuits or issues for the
Post
.

When I mentioned in an editorial meeting in early 2009 that 3FN had emerged as the central focus of a U.S. law-enforcement investigation into cybercrime, it was strongly suggested that we get confirmation of that fact from at least two sources, or wait for an on-the-record law-enforcement comment about the investigation or for proof of legal proceedings to be filed against the hosting provider before moving forward with any story alleging badness at 3FN. The case documents had been sealed by a federal judge, and my law-enforcement source was the only one I knew who had even heard of 3FN. The story was held.

Then, on June 2, 2009, more than fifteen thousand websites hosted at 3FN were yanked offline after the U.S. Federal Trade Commission (FTC) convinced a Northern California district court judge to have the company’s upstream Internet providers stop routing traffic for the provider. The FTC alleged that 3FN operated “as a ‘rogue’ or ‘black hat’ Internet service provider that recruited, knowingly hosted, and actively participated in the distribution of illegal, malicious, and harmful content,” including botnet control servers, child pornography, and rogue antivirus products.

The FTC’s action provided the backstopping I needed to finally gather sufficient support to move ahead with my investigation into Vrublevsky, ChronoPay, and their role in fostering the fake antivirus market that was plaguing millions of consumers and threatening their identities, finances, and security. Crutop.nu also was hosted at 3FN and was even named in the FTC’s action. The FTC called Crutop a place “where criminals share techniques and strategies with one another” and a Russian language website “that features a variety of discussion forums that focus on making money from spam.” A review of multiple discussion threads at the Russian adult webmaster forum indicated that Crutop’s more than eight thousand active members had been 3FN’s single largest customer base.

Tellingly, directly after 3FN was taken down—but before washingtonpost.com ran the story on ChronoPay’s ties to the rogue antivirus industry—Crutop.nu’s homepage was changed to a lengthy screed about the FTC’s action against 3FN. This would be my first introduction to Vrublevsky’s epic rants. The message read, in part:

And in conclusion we would like to add, that while paragraph 1 of our rules has never been taken seriously before and was written as a joke, but related to recent events we would like to know how it was possible that five (5!) reputable experts-agents (including NASA experts and Mr. Brian Krebs) from the USA (where every tenth person speaks Russian, source: Wikipedia), could not figure out that on Crutop.nu in the SPAM sub-forum, discussions have nothing to do with mail spam or other cybercrimes?

The story on Vrublevsky and ChronoPay’s key role in 3FN finally ran more than four months after I turned it in. No lawsuit from him or ChronoPay followed. But the editors at the
Washington
Post
said they were still deeply concerned about my focus on Internet bad guys. The
Post
higher-ups were nervous about my reporting on a crime-heavy
subject in which the standard forms of documentary evidence don’t typically exist. Also, they took the position that my focus on cybercrime—as opposed to a broader beat such as consumer technology or technology policy—was too narrow, and that I was getting too close to my sources to remain objective.

I shared their concerns—to a degree. No journalist wants to depend on a handful of sources to the exclusion of others; doing so risks publishing stories that lack perspective and balance. But I knew the solution here was that I merely needed more and better sources—particularly those actively engaged in the cybercrime community. I also was convinced that the 3FN story was too important not to pursue, and that setting the story aside would be a waste of a good opportunity to expose—and potentially stop—a great deal of cybercrime activity.