| <!doctype html> |
| <html class="no-js" lang="en" dir="ltr"> |
| <head> |
| <meta charset="utf-8"> |
| <meta http-equiv="x-ua-compatible" content="ie=edge"> |
| <meta name="viewport" content="width=device-width, initial-scale=1.0"> |
| <title>DDOS mystery involving Linux and mod_ssl - Apache Infrastructure Website</title> |
| <link href="/css/bootstrap.min.css" rel="stylesheet"> |
| <link href="/css/fontawesome.all.min.css" rel="stylesheet"> |
| <link href="/css/headerlink.css" rel="stylesheet"> |
| <script src="/highlight/highlight.min.js"></script> </head> |
| <body class="d-flex flex-column h-100"> |
| <main class="flex-shrink-0"> |
| <!-- nav bar --> |
| <nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example"> |
| <div class="container-fluid"> |
| <a class="navbar-brand" href="/"><img src="/images/feather.png" style="height: 32px;"/> Apache Infrastructure</a> |
| <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation"> |
| <span class="navbar-toggler-icon"></span> |
| </button> |
| |
| <div class="collapse navbar-collapse" id="navbarADP"> |
| <ul class="navbar-nav me-auto mb-2 mb-lg-0"> |
| <li class="nav-item dropdown"> |
| <a class="nav-link dropdown-toggle" href="#" data-bs-toggle="dropdown" aria-expanded="false">About</a> |
| <ul class="dropdown-menu"> |
| <li><a class="dropdown-item" href="/team.html">About the team</a></li> |
| <li><a class="dropdown-item" href="/roundtable.html">The Infrastructure Roundtable</a></li> |
| <li><a class="dropdown-item" href="/blog/">The Infrastructure Blog</a></li> |
| </ul> |
| </li> |
| <li class="nav-item"> |
| <a class="nav-link" href="/policies.html">Policies</a> |
| </li> |
| <li class="nav-item dropdown"> |
| <a class="nav-link dropdown-toggle" href="#" data-bs-toggle="dropdown" aria-expanded="false">Services and Tools</a> |
| <ul class="dropdown-menu"> |
| <li><a class="dropdown-item" href="/services.html">Services and Tools</a></li> |
| <li><a class="dropdown-item" href="/machines.html">Machines and Fingerprints</a></li> |
| <li><a class="dropdown-item" href="https://blocky.apache.org/">Blocky</a></li> |
| <li><a class="dropdown-item" href="https://app.datadoghq.com/account/login?next=%2Finfrastructure">DataDog</a></li> |
| <li><a class="dropdown-item" href="https://whimsy.apache.org/roster/committer/" target="_blank">Committer Search</a></li> |
| </ul> |
| </li> |
| <li class="nav-item dropdown"> |
| <a class="nav-link dropdown-toggle" href="#" data-bs-toggle="dropdown" aria-expanded="false">Documentation</a> |
| <ul class="dropdown-menu"> |
| <li><a class="dropdown-item" href="/doc.html">Contribute</a></li> |
| <li><a class="dropdown-item" href="/infra-volunteer.html">Volunteer with Infra</a></li> |
| <li><a class="dropdown-item" href="/how-to-mirror.html">Become an ASF download mirror</a></li> |
| <li><a class="dropdown-item" href="/hosting-external-agent.html">Host a Jenkins or Buildbot agent</a></li> |
| |
| </ul> |
| </li> |
| <li class="nav-item"> |
| <a class="nav-link" href="/stats.html">Status</a> |
| </li> |
| <li class="nav-item"> |
| <a class="nav-link" href="/contact.html">Contact Us</a> |
| </li> |
| </ul> |
| </div> |
| </div> |
| </nav><!-- breadcrumbs --> |
| <div class="card" style="height: 34px;"> |
| <nav aria-label="breadcrumb" style="padding-left: 12px; padding-top: 4px;"> |
| <ol class="breadcrumb"> |
| <li class="breadcrumb-item"><a href="/">Home</a></li> |
| |
| <li class="breadcrumb-item active"><a href="/blog/ddos_mystery_involving_linux_and.html"> |
| DDOS mystery involving Linux and mod_ssl </a></li> |
| |
| <li class="breadcrumb-item active">(<a href="https://github.com/apache/infrastructure-website/tree/master/content/blog/ddos_mystery_involving_linux_and.md">edit</a>)</li> |
| |
| </ol> |
| </nav> |
| </div> |
| |
| |
| <!-- page contents --> |
| <div id="contents"> |
| <div class="bg-white p-5 rounded"> |
| <div class="col-sm-8 mx-auto"> |
| <h1> |
| DDOS mystery involving Linux and mod_ssl |
| </h1> |
| <p>Posted on: 2009-10-12 01:53:03+00:00</p> |
| <p>In the first week of October we started getting reports of performance issues, mainly connection timeouts, on all of our services hosted at <a href="https://issues.apache.org" title="https://issues.apache.org/">https://issues.apache.org/</a>. On further inspection we noticed a huge amount of "Browser disconnect" errors in the error log right at the beginning of the ssl transaction, on the order of 50 connections / second. This was grinding the machine to a standstill, so we wrote a quick and dirty <a href="http://people.apache.org/~joes/ddos_accept.pl">perl script</a> to investigate the matter. Initial reports indicated a ddos attack from nearly 100K machines targeting Apache + mod_ssl's accept loop, and the script was tweaked to filter out that traffic before proxying the connections to httpd.</p><p>As we started getting a picture of the IP space conducting the attack, the prognosis looked rather bleak: more and more IP's were getting involved and the ddos traffic continued to increase, getting to the point where Linux was shutting down the ethernet interface. So we then rerouted the traffic to an available FreeBSD machine, which did a stellar job of filtering out the traffic at the kernel level. We unfortunately didn't quite realize how good a job FreeBSD was doing, and for a time we were operating under the impression that the ddos was ending. So we eventually moved the traffic back to brutus, the original Linux host, and <a href="http://people.apache.org/~joes/avoid_dos_2.2.x-try2.diff">patched httpd</a> using code developed by Ruediger Pluem.<br/></p><p>And back came the ddos traffic. In a few days the rate of closed connections had nearly doubled, so we had little choice but to start dumping the most frequent IP addresses into iptables DROP rules. 5000 rules cut the traffic by 2/3 in an instant. But the problem was growing- our logs indicated there were now over 300K addresses participating in the attack.</p><p>We started looking closer at the IP's in an attempt to correlate them with regular http requests. The only pattern that seemed to emerge was that many of the IP's in question we're also generating spartan "GET / HTTP/1.1" requests with a single Host: <a href="http://140.211.11.140">140.211.11.140</a> header to port 443. Backtracking through a year of logs revealed that these spartan requests had been going on since August 6, 2008. The IP's originating these requests were as varied as, and more often that not matched up with, the rapid closed connection traffic we started seeing in October.<br/></p><p>So what exactly is going on here? The closed connection traffic continues to rise, and the origin of the associated spartan requests is currently unknown.</p> |
| |
| </div> |
| </div> |
| </div> |
| <!-- footer --> |
| <div class="row"> |
| <div class="large-12 medium-12 columns"> |
| <p style="font-style: italic; font-size: 0.8rem; text-align: center;"> |
| Copyright 2024, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/> |
| Apache® and the Apache feather logo are trademarks of The Apache Software Foundation... |
| </p> |
| </div> |
| </div> |
| <script type="application/ecmascript" src="/js/bootstrap.bundle.min.js" integrity="sha384-OERcA2EqjJCMA+/3y+gxIOqMEjwtxJY7qPCqsdltbNJuaOe923+mo//f6V8Qbsw3"></script> </main> |
| </body> |
| </html> |
