.comment: The Great, the Pretty Bad, and the Breathtakingly Stupid - page 3
I very much dislike being lied to, but I like it even less when the prevaricators do not even show me the respect of coming up with a plausible lie. Which is another way of saying that I've not enjoyed my conversations over the last three weeks with Charter Communications, who provides cable modem service here. The saying is that one should never ascribe to malice that which can be explained by stupidity, but in this case we seem to have both: they have, through their guy Dave (who was answering the phone there Monday night when I phoned for that day's excuse for why their cable modem service was still down), said that they were lying Friday when they said that service would be in good shape by Monday; and my dealings with them are such that I would characterize most of the people there as blithering idiots but for my lack of confidence in their ability to blither.
You may have gathered by now that I am a little angry with Charter Communications. So let me tell you about it.
On July 19, in the afternoon, the Charter Communications cable modem service started to go flaky. The cable modem (a Motorola SB3100, if it matters) got hammered so heavily that it shut down and recycled a couple of times. The activity continued late into the evening, and in breaks in the storm I learned about something called Code Red.
For the next week and a half there was a little more activity than usual, but there are always people doing port scans, and I keep things buttoned up here as tightly as I can. I don't run any servers here (except for my own outgoing smtp), so no one has any business coming in, and I do my best to turn tourists away. Then, on August 1, the activity picked up again, and the modem recycled once that I saw. This became pretty much the way of things.
I phoned Friday and asked what they were doing to deal with the problem. They told me this: "There's noise on the line, and our people are out in the field round the clock trying to track it down. It's difficult to fix because some places are affected more than others, even though they're on the same line. They told us they hoped to have it fixed today, then tomorrow; now they say it won't be fixed for sure until Monday." On a whim, I did a Netcraft Uptime lookup of chartermarketplace.com, a thing the Charter peppers every television show with advertisements about. I was unsurprised that the site, though in California and not here in Connecticut, is Windows 2000 running IIS 5.0.
Saturday. Ah, yes, Saturday. Though you won't hear about it on television (it has no direct effect, after all, on the search for Chandra Levy), Saturday began what was arguably the most fiendish attack in more than a decade on the Internet. Dubbed Code Red version 2, the new contagion is not, contrary to reports, at all like the original worm but for its use of much the same exploit. Vulnerable machines are unpatched Windows 2000 boxen running IIS. There appears to be a lot of them.
Unlike the first Code Red, CRv2 creates a backdoor. It is not repaired by applying the patch Microsoft cranked out after eEye discovered the exploit that made Code Red possible (which kind of forced Microsoft to admit there was a problem, though they didn't accent this when they put out the patch, which may have played a part in why so few people had applied it when the first Code Red hit). Instead of the original 100 threads looking for other victims, this one throws either 300 or 600 depending on where it is located. And instead of just throwing scans at Port 80 of random IP addresses, it spends half or more of its time on nearby addresses. You can imagine the effect this had on, say, cable providers who had scads of users who either didn't know they had IIS running or else who were hosting their own little vanity websites, contrary to their contracts with the cable company.
CRv2 was initially very bad, but it got worse and worse. It thoroughly infiltrated the @home network in the northeast. It hammered AT&T's Multiple Protocol Network. It effectively brought down much of IBM's network. It got into Intel and, last I heard, was still there. As a pure DoS attack, it was and is impressive, and its effects were and are widespread.
At the time of CRv2's arrival, the cable modem here, attached to the cable system of Charter Communications, began to go crazy. The modem began to crash a couple times per hour. When it wasn't down, the storm outside my firewall was so fierce that throughput was in the neighborhood of 6kbps on a 512k connection. I had to tell sendmail to keep trying, and frequently, to send the messages stacking up in mqueue, in hope of getting at least some of them through during the occasional lulls when the modem also happened to be working. By Sunday evening, the modem was down more than it was up -- it would come back only for a minute or two.
First thing Monday morning, the modem still highly intermittent and throughput all but nonexistent on the rare occasions there was any at all, I phoned Charter Communications. This involved hitting redial for 10 minutes straight, then 20 minutes listening to music so bad they wouldn't even use it as background noise on a tornado chaser show. Finally an actual person, and she was not happy at being bothered by mere customers.
"The problem is line noise, and they've been working on it for two weeks," she said. "It has nothing to do with Code Red." (This was, interestingly, in stark contrast to what Cox Communications was telling its customers -- that they'd been whacked good and hard by Code Red 2 and were still trying to figure out what to do about it.)
Monday night I phoned for the evening's excuse and got the fellow Dave. He, too, held to the notion that it was just "line noise. I wish I could blame it on Code Red, but I can't," he said. The fact that the drop in service exactly corresponded with Code Red and CRv2 was, I guess, a mere coincidence, as was the scanning of port 80 at as much as 500 times per minute? "That's us," he said. "We're trying to hook back up to the modems." By scanning port 80? "Yes." But port 80 is used only by web servers, which your contract specifically prohibits. "Well, that's what we're doing and have been doing for the last two weeks." (Funny thing is, I half believe him -- the port 80 traffic very well could be generated by machines owned by Charter Communications, though for reasons other than the one given.)
What about the story I was told Friday, that this would definitely be fixed by Monday? "They were lying to you." Well, yes, that does seem to be something that one does encounter there. "But it will definitely be fixed by the end of the week," poor Dave said. Which I am sure is what someone who got his MSCE at one of the very best matchbook correspondence schools told him.
Here's what I think happened:
They got well and truly blistered by Code Red, all flavors and name-alikes. Embarrassed, possibly by their own unpatched machines but even more likely by those of customers, especially business customers, they lied. They figured that everybody on their system was dimwitted enough to believe them. Now they're in a fix and don't know what to do about it.
Here's how they could solve the problem:
Block everything coming into the network aimed at any port 80 on any machine on the network. All of it. Permanently. Then it's a simple matter to determine which machines on the network are generating the traffic locally. Take 'em off the network and telephone them to explain why, and that with the current infection, applying Microsoft's patch won't solve the problem. Rebooting won't solve the problem. Send 'em a sheet -- it's only about a page long -- explaining how to get rid of the thing. (Though with the backdoors in place and the infected machines announcing all across the great wide web who they are, no infected machine can be considered secure without a complete reformat and reinstall, because anyone could have done anything to the machine in the interim.) When the user can demonstrate that the machine has been secured, it will be allowed back on the network, but only then. And only with the understanding that if the owner wants a web site, there's the need to contract with a real web hosting company. There are a lot of them, some very good, and even the good ones don't cost all that much.
And with the further understanding that next time, the disconnection will be permanent.
The same thing would have to apply, too, to commercial clients, with the lone difference that they get port 80 access. This would involve segregating them in their own IP space, on, really, their own little network. Regrettable, but in the final analysis, tough: If you drive recklessly, there's an increased chance you'll end up in a wheelchair. Regrettable, certainly. But the price that is paid for reckless behavior. Places like Charter and @home have known for a long time, unless they really are sub-blitherers, that this kind of thing was going to happen and that their networks (and many people on them) were running unsecure boxen and they did nothing, or at least nothing effective (kind of like the Charter people who have been wandering "in the field" for almost three weeks now, instead of back at the office fixing their screwed-up system). Now the price will be paid, to avoid an even higher price later.
I'm not sure whether the people at Charter actually themselves believe that their mythical untraceable "line noise" and the Code Red and CRv2 worms are one of those rare, perfectly-parallel-at-thousands-of data-points, coincidences that we never hear about because they never happen -- look at the odds of picking even five "data points" that are the same ones announced on teevee by the lottery guy.
It all reminds me of a Lily Tomlin piece on Saturday Night live back when it was still funny. She was doing her telephone operator schtick, with the closing slogan, "We don't care. We don't have to. We're the phone company."
So it always is with monopolies, as the recent worm business has illustrated here twice, involving two different companies. I may well have to FedEx this to my editor for it to achieve its usual Wednesday appearance. But those two companies don't care. They don't have to. They're Microsoft and Charter Communications.