An Astonishing Collaboration
Wow. It’s out. It’s finally, finally out.
Sweet!
So there’s a bug in DNS, the name-to-address mapping system at the core of most Internet services. DNS goes bad, every website goes bad, and every email goes…somewhere. Not where it was supposed to. You may have heard about this — the Wall Street Journal, the BBC, and some particularly important people are reporting on what’s been going on. Specifically:
1) It’s a bug in many platforms
2) It’s the exact same bug in many platforms (design bugs, they are a pain)
3) After an enormous and secret effort, we’ve got fixes for all major platforms, all out on the same day.
4) This has not happened before. Everything is genuinely under control.
I’m pretty proud of what we accomplished here. We got Windows. We got Cisco IOS. We got Nominum. We got BIND 9, and when we couldn’t get BIND 8, we got Yahoo, the biggest BIND 8 deployment we knew of, to publicly commit to abandoning it entirely.
It was a good day.
CERT has details up, and there’s a full-on interview between myself and Rich Mogull up on Securosis. For the non-geeks in the audience, you might want to tune out here, but this is my personal blog and I do have some stuff to mention to the crew.
There’s something very important about what we accomplished here.
We. Because there’s absolutely no way I could have pulled this off by myself.
Paul Vixie is an institution. Having long maintained the Internet’s most popular DNS server, Paul simply knows everybody. Paul was absolutely instrumental in pulling together the engineers we needed to solve this problem. We needed Florian Weimer there, all the way from Europe. We needed David Dagon, and Jinmei Tatuya, and Wouter Wijngaards. We needed Microsoft, Cisco, Nominum, Neustar, and OpenDNS.
And we really needed CERT.
It was an interesting discussion, with lots of disagreement, but ever-growing consensus. After evaluating several options, one approach was clear — and, I must admit, somewhat embarassing to Paul.
DJB was right. All those years ago, Dan J. Bernstein was right: Source Port Randomization should be standard on every name server in production use.
There is a fantastic quote that guides a lot of the work I do: Luck is the residue of design. Dan Bernstein is a notably lucky programmer, and that’s no accident. The professor lives and breathes systems engineering in a way that my hackish code aspires to one day experience. DJB got “lucky” here — he ended up defending himself against an attack he almost certainly never encountered.
Such is the mark of excellent design. Excellent design protects you against things you don’t have any information about. And so we are deploying this excellent design to provide no information.
To translate the fix strategy into a more familiar domain, imagine large chunks of Windows RPC went from Anonymous to Authenticated User only, or even all the way to Admin Only. Or wait, just remember Windows XPSP2 
This is a sledgehammer, by design. It cuts off attack surface, without necessarily saying why. Astonishingly subtle bugs can be easily hidden, or even rendered irrelevant, by a suitably blunt fix.
Of course, it remains obvious that something new is up, and that something will be found eventually. But there’s a lot of buggy systems out there, vulnerable not just to new bugs but bugs that have been known for years. If all this effort ever accomplished was sweeping old and crusty BIND8 off the Internet, if we could finally fully eliminate Joe Stewart’s (edit: Originally Vagner Sacramento’s, thanks Joe!) Birthday Attacks from 2002, if we started doing something about Amit Klein’s Transaction ID Randomness finds (even the deeply underrated client vulns) from last year, and yes, if the static port assignments DJB warned us about ages ago were finally shut down — then this would still be a huge win.
There are reasons why the new issue is particularly severe, but I think reasonable people can agree that anything that could scrub even the old bugs would be a boon to the security of the Internet. And so, I ask the open research community…assume I found nothing! Assume this is nothing but a stunt, to finally get people to take Joe and Amit and DJB seriously, and to give network scanners a crystal clear fingerprint of what a trustable recursive server looks like.
Joe and Amit and especially DJB have done some incredible work. I’d look terrible at the end of it, but their bugs would finally get fixed, and stay fixed. As for me, I dunno. Go back to graphics Mmmm…SIGGRAPH…
For those of you who won’t make that assumption, I have a request. It is very unusual, and maybe unreasonable. But I have to ask.
I want you to explore DNS. I want you to try to build off the same bugs I did to figure out what could possibly go wrong. Maybe I missed something — I want you to help me find out if I did, so we can deal with it now instead of later. I do want all this. But I also want my family to be able to use the Internet in peace. I’m not asking for forever. I am asking about thirty days. I’ve done everything in my power to get the patches available, no matter the platform. But the code doesn’t (always) install itself. While I’m out there, trying to get all these bugs scrubbed — old and new — please, keep the speculation off the @public forums and IRC channels. We’re a curious lot, and we want to know how things break. But the public needs at least a chance to deploy this fix, and from a blatantly selfish perspective, I’d kind of like my thunder not to be completely stolen in Vegas
Now, if you do figure it out, and tell me privately, you’re coming on stage with me at Defcon. So I can at least offer that
An amazing find…carry on the good work, you deserve your thunder.
Speak soon,
Hanif
Question: What IS your email/contact info for our random speculation?
When I try your checker (in FF3; on debian, FWIW), I get a 404 error page in the box, it seems to be failing to load 853784753500.toorrr.com/printme.html (exact hostname varies)
Does this mean I’m vulnerable, not vulnerable, or on an unsupported platform? Or something else strange going on.
I’m just wondering, so you found all this bug on all these diff platforms/OS’s, i can’t stop wondering… to be able to find all that, how many times did you use the bug in order to gain access on each of those platforms/OS’s to be sure that the bug is real and not just another fake seemingly flaw ? And if so of which side that counts? White Hat or Black Hat ?:) I’m pretty sure there were plenty people whom knew about this bug, found it, use it to gain access and do silly things and that’s how it was spotted.Atleast that’s kinda logical to me.
But if you’re saying that you changed the course of history by doing this, meaning the Security was first to find the bug and not the actual Hackers, then that’s a First. Correct me if i’m wrong but i never found 1, ONE ! person whom found a bug and didn’t use it. I know back in the old days when a wellknown(these days) person found a major bug, use it to gain access on all possible or imaginable “spots” before he actually reported the bug and provided the fix for it. How are you different from any other ?
Thanks for the massive efforts ;>
Thank you.
I just hope BIND admins aren’t as clueless as RedHat.
Their updated BIND and caching-nameserver RPMs still have example (and real) .conf files which contain the following:
query-source port 53;
query-source-v6 port 53;
Even a patched BIND will still use a fixed port if those lines are in the config.
Ooops….
Tried Security Update for Windows Server 2003 (KB951746) and the DNS vulnerability scanner on doxpara.com says I’m still vulnerable?
RedHat have released an updated advisory, fixing the issue I mentioned above, and warning admins to check their configs.
Yay!
I apologize for the cross-posting, Dan, but can you address this:
I’m also interested in the comments highlighted by Tom Cross from IBM/ISS X-force reiterating a comment sent to FD regarding DNS and NAT devices. You can find Tom’s post here: blogs.iss.net/archive/dnsnat.html
” I’m writing this blog post in order to draw more attention to that observation, as I’m not sure that the security industry has realized its full implications. As far as we know, this observation applies to any NAT device, not just the firewall imipack tested, and we think it has significant implications for network architecture.
When a host on a computer network selects a source port for a UDP request, it selects a port that is unique for that host. When a one-to-many hiding NAT device receives that request and translates it, it has to assign a new source port, because the NAT device has to assign unique ports for all of the hosts on the internal network. X-Force is not aware of any NAT device on the market that randomly selects UDP source ports. Therefore, when patched DNS clients and servers are used behind NAT, they are still vulnerable to attack. The source port entropy introduced by the recent patches is cancelled out by the NAT device.
It is our opinion that network administrators who have caching internal DNS servers behind NAT should plan to move those servers into a DMZ where they can be directly assigned a unique Internet IP address. Any servers that remain behind NAT devices will remain vulnerable after patches have been applied, and X-Force suspects that given the amount of media attention DNS Cache Poisoning has received this week that an increased frequency of attack is likely in the coming months.”
What say thee to that? Patching may not be enough if you physically have to move DNS servers to what amounts to DMZ’s and then expose them with 1-to-1 NAT’s instead of hides…oh the fun.
/Hoff
Really DNS, are importants in the Web, than it could provocate many damages to people.
Specially in my country, DNS are full, because the capacity is for only a amount, but we have exceed it.
visiong12peru.blogspot.com/
DJB was lucky? I think not… he speculated a future security issue.
I’ve wondered for a while about something. Can a DNS request request the resolution of more than one domain name? If so, it seems like you could ask remote domain servers to resolve both “www.google.com” and “7fe85813f1d6d7cc.bogus” for you, with the rejection message acting as proof of knowledge of the nonce.
How to resolve the infection (Cache Poisoning)?
Wow. If that’s truth it’s impressive.
Random ports? Hmmmmmm…………
Thanks for your dns resolver checker! I’ll note that I do indeed see non-randomness in the source port selections in my current dns resolver that the simple check script didn’t notice, so beware folks.
But it would be much more useful if we could run it against arbitrary dns servers, and from remote hosts. It uses javascript for both the request and as part of the first response and javascript often isn’t included in text-mode browsers. How about a python or perl version?
Thanks again, and congrats on such a well-rolled-out announcement.
Ian wrote:
> Tried Security Update for Windows Server 2003
> (KB951746)
Good.
> and the DNS vulnerability scanner on
> doxpara.com says I’m still vulnerable?
This indicates that your ISP still needs to update their systems. (Or another ISP along the chain that your local ISP uses for access.)
End users are generally not the target of DNS poisoning, except maybe on your local Wi-Fi or college network. The usual target would be the DNS system of a large ISP or corporate.
Joey Hess wrote:
> When I try your checker (in FF3; on debian, FWIW), I get a 404
> error page in the box, it seems to be failing to load
> 853784753500.toorrr.com/printme.html (exact hostname
> varies)
Something between you and toorrr.com is dropping packets. Possibly accidental or possibly policy from your ISP, (or college/employer if you are not at home).
ttfn,
TonyN
good
Tried Security Update for Windows XP SP2 (KB951746) and the DNS vulnerability scanner on doxpara.com says I’m still vulnerable…
DJB’s track record on writing secure, solidly-designed software is the reason that I STILL run qmail today.