| title: DDOS mystery involving Linux and mod_ssl |
| date: '2009-10-12T00:00:00+00:00' |
| <p>In the first week of October we started getting reports of performance issues, mainly connection timeouts, on all of our services hosted at <a href="https://issues.apache.org" title="https://issues.apache.org/">https://issues.apache.org/</a>. On further inspection we noticed a huge amount of "Browser disconnect" errors in the error log right at the beginning of the ssl transaction, on the order of 50 connections / second. This was grinding the machine to a standstill, so we wrote a quick and dirty <a href="http://people.apache.org/~joes/ddos_accept.pl">perl script</a> to investigate the matter. Initial reports indicated a ddos attack from nearly 100K machines targetting Apache + mod_ssl's accept loop, and the script was tweaked to filter out that traffic before proxying the connections to httpd.</p><p>As we started getting a picture of the IP space conducting the attack, the prognosis looked rather bleak: more and more IP's were getting involved and the ddos traffic continued to increase, getting to the point where Linux was shutting down the ethernet interface. So we then rerouted the traffic to an available FreeBSD machine, which did a stellar job of filtering out the traffic at the kernel level. We unfortunately didn't quite realize how good a job FreeBSD was doing, and for a time we were operating under the impression that the ddos was ending. So we eventually moved the traffic back to brutus, the original Linux host, and <a href="http://people.apache.org/~joes/avoid_dos_2.2.x-try2.diff">patched httpd</a> using code developed by Ruediger Pluem.<br /></p><p>And back came the ddos traffic. In a few days the rate of closed connections had nearly doubled, so we had little choice but to start dumping the most frequent IP addresses into iptables DROP rules. 5000 rules cut the traffic by 2/3 in an instant. But the problem was growing- our logs indicated there were now over 300K addresses participating in the attack.</p><p>We started looking closer at the IP's in an attempt to correlate them with regular http requests. The only pattern that seemed to emerge was that many of the IP's in question we're also generating spartan "GET / HTTP/1.1" requests with a single Host: <a href="http://140.211.11.140">140.211.11.140</a> header to port 443. Backtracking through a year of logs revealed that these spartan requests had been going on since August 6, 2008. The IP's originating these requests were as varied as, and more often that not matched up with, the rapid closed connection traffic we started seeing in October.<br /></p><p>So what exactly is going on here? The closed connection traffic continues to rise, and the origin of the associated spartan requests is currently unknown.</p> |
