Date: March 19, 2017
Just weeks after a headline-grabbing Internet infrastructure attack, the Internet Architecture Board (IAB) presented a timely and engaging technical talk to IETF 97 participants aimed at sharing the vulnerabilities associated with the massive cyberattack, as well as potential fixes that the standards body can pursue.
The IAB was preparing a different technical talk for IETF 97, but scrapped its plans after a large-scale Distributed Denial of Service (DDoS) attack harmed managed DNS infrastructure provider Dyn on 21 October 2016. Two aspects of the Dyn attack were unusual: first, the attack came from a botnet comprising Internet of Things (IoT) devices; and second, it was the largest attack of its kind in history.
The Dyn DDoS attack came in three waves over a six-hour period. Although Dyn says it never suffered a network-wide outage, the company’s managed DNS infrastructure was slowed to the point where many of its customers—including marquee Internet brands such as Twitter, CNN, and Netflix—were unreachable.
In addition to creating a massive Internet disruption for customers, what made the Dyn DDoS attack newsworthy is that it involved tens of thousands of discrete IP addresses from the Mirai botnet, which comprises IoT devices. This foreshadows a future where more attacks are derived from IoT devices that are typically not secured or even regularly upgraded.
“We are here to talk about a new class of attacks on the Internet architecture—or perhaps just some insights we hope are new,” said IAB member Suzanne Woolf, who introduced the technical plenary session. She said the Dyn DDoS attack “caused an explosion of attention on DNS, Internet of Things, mass compromise of Internet-connected devices, and the business and operational models underlying the provisioning of content at Internet scale.”
The first speaker was Nick Sullivan, head of cryptography at Cloudflare and an active participant in the IETF’s work related to Transport Layer Security (TLS). He said what was unique about the Dyn DDoS attack was its magnitude.
“This is not something particularly new. Botnets exist and have existed for a long time,” Nick said. “But this is an example of botnets at a larger scale, and they sent a mix of attacks.”
Nick said the majority of DDoS attacks seen on the Internet in 2016 were known attacks—DNS floods, Syn floods and HTTPS floods—but that these attacks are now coming at a larger scale.
DNS attacks fall into two camps: direct attacks on the authoritative DNS server from a botnet or botnet attacks that go through valid recursors before they move to the authoritative DNS server.
For direct attacks, Nick recommended that DNS operators treat every request from an unknown resolver with suspicion and assume that a flood of requests from nonresolvers to an authoritative server is an attack. “Just drop the packets,” he advised.
Nick said these floods are typically requests for an Apex Domain or random subdomains. He added that sometimes these floods come with spoofed source addresses, which are harder to handle.
“The Mirai botnets often are not spoofed addresses, which makes them slightly easier to deal with,” he said.
When experiencing a DNS flood, you shouldn’t advertise the IPs that are under attack as what’s called “null-routing IPs,” because it would cause them to fall off the Internet, Nick said. “This is a pretty dangerous thing to do, and it has lots of repercussions,” he said. “Lately, attacks have been attacking entire subnets and that makes this entirely unreasonable as a defense mechanism.”
Instead, Nick suggested spreading the load of the attack geographically with the Anycast protocol and across data centers with the equal-cost multipath (ECMP) routing protocol. He also recommended filtering packets as early as possible. “The main way to handle a large flood is to make sure that only legitimate applications get to your application,” he said.
Nick recommended filtering traffic through iptables Berkeley Packet Filter (BPF) rules, which are powerful techniques for matching and blocking packets inside the kernel and can be automated. He admitted that the iptables BPF rules must be dynamic and use machine learning and heuristics to keep up with ever-changing attack profiles. “The attacks you see from one botnet are not what you’re going to see from another, so this has to be very dynamic,” he said. “If you have static rules, you are probably going to go down. If possible, move the rules outside of your server and into the Network Interface Card. This can help relieve the load.”
Nick said that DNS Floods that go through recursive servers are different types of attacks, and that network operators should answer these attacks rather than try to block and drop the traffic. That’s because recursive DNS is making requests for legitimate customers at the same time as attackers. “If it’s possible to whitelist known recursive DNS servers, do so,” he recommended. “This is very helpful and the right thing to do.”
He advised against rate limiting this traffic as it can cause negative effects including amplifying the attack. “Rate limiting is not a very effective method for this. Really, you have to handle the packets,” he said.
One suggestion is taking advantage of NSEC, which is a feature of the DNS Security Extensions (DNSSEC) protocol that is used to prove a name doesn’t exist. “Potentially caching this for nonsigned ranges might help,” Nick suggested. “It’s one of the many different options for keeping the traffic from going all the way through to the authoritative server.”
With regard to battling Syn flood attacks, he said using BGP Anycast for TCP and establishing dynamic IPtables BPF rules can be effective.
For HTTPS floods, you can use the protocol’s rate limiting features, which include rate limiting by request or volume. He added that “a simple TCP reset will get you a long way.”
Nick said that DDoS attacks are sent by botnets consisting of compromised endpoints, which are increasingly IoT devices. “We are in the early days of this new set of devices that are running software, and the software is getting old on a lot of these devices,” he said, warning that the IoT-based botnet attacks are likely to get worse.
He explained that IoT devices are low-cost, low-margin devices and that manufacturers are not incentivized to build in security or even a method to upgrade the devices later in response to newfound vulnerabilities.
“Nothing about these attacks should be new to you,” Nick told the audience. “We’re dealing with the same problems, but now at a much larger scale, and it’s exposing certain things about the Internet of Things that we’ve known.”
The best advice that Nick offered was for network operators to stop bad traffic as close as possible to the entry point and to use Anycast. “Distribute the load and filter early,” he advised.
He also noted that being able to stay online while experiencing a DDoS attack requires scale.
“You have to be big, you have to be smart, and you have to have the tools and to work together with other people to stop the attacks as close as possible to the edge,” he concluded.
The second speaker at the technical plenary session was IAB member Andrew Sullivan, who also serves as director of DNS engineering at Dyn. Andrew said that the attack against Dyn’s infrastructure was large in scale and that even the company’s well-built Anycast-based DNS system couldn’t withstand it without long latency and resolution failures.
“We’ve seen lots of standard amplification attacks,” Andrew said. “This particular attack had a high proportion of TCP—we don’t normally see this. This was comparatively low spoofing. We have definitely confirmed 40,000 addresses involved in the botnet, although it may be up to 100,000.”
Andrew said he doesn’t believe that the Dyn DDoS attack was a one-time occurrence.
“There was a previous attack just a couple weeks before this that was very, very large,” he warned. “We know the Mirai botnet code is out there and is being improved upon by various people.”
Andrew said attacks of this type are ironic because they use the strengths of the Internet—its distributed nature, ease of attaching endpoints, and lack of intelligence in the network itself—to attack the Internet architecture. He said that the endpoints are supposed to be more intelligent than the network itself, but that this design philosophy isn’t true with IoT systems.
“It’s obvious that Internet of Things is going to continue to create these types of problems,” Andrew said. “It creates ubiquitous connectivity and very, very lightweight systems… If you have millions and millions of these devices all over the place, you will have lots of compromised hosts.”
Andrew fears that attacks like the recent Dyn DDoS attacks could result in government regulations aimed at preventing compromised hosts from connecting to the Internet, which would reduce the openness of the Internet.
“We are the people who ought to tackle this problem because we understand the technology, we understand the incentives, and we understand the nature of the underlying architecture,” Andrew concluded. “I don’t have a magic solution, but I hope we can have an interesting and useful discussion.”
This technical talk was followed by a lively question-and-answer period that raised such suggestions as creating lightweight security protocols for IoT devices and otherwise advising endpoint manufacturers on how to build in security at a lower cost. The goal of these suggestions was for the IETF to find ways to make it easier and cheaper for people building IoT devices to have security by default.
At the panel’s conclusion, Suzanne said the IAB would continue this discussion online.
In other news, the Internet Society at this plenary session presented its Jonathan B. Postel Service Award to Kanchana Kanchanasut for her pioneering work in establishing Internet services in her native Thailand and across Southeast Asia. Kanchanasut receives a crystal trophy and US$20,000. She is the 19th recipient of the award, which has been given since 1999.