| <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" |
| "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> |
| |
| <html xmlns="http://www.w3.org/1999/xhtml"> |
| <head> |
| <meta name="generator" content="HTML Tidy, see www.w3.org" /> |
| |
| <title>Running a High-Performance Web Server for BSD</title> |
| </head> |
| <!-- Background white, links blue (unvisited), navy (visited), red (active) --> |
| |
| <body bgcolor="#FFFFFF" text="#000000" link="#0000FF" |
| vlink="#000080" alink="#FF0000"> |
| <a id="initial" name="initial"> |
| <!--#include virtual="header.html" --> |
| </a> |
| |
| <h1 align="CENTER">Running a High-Performance Web Server for |
| BSD</h1> |
| |
| <p>This document assumes that you have read the appropriate |
| overview documentation for |
| <a href="http://www.FreeBSD.org/docs.html">FreeBSD</a>, |
| <a href="http://www.NetBSD.org/Documentation/">NetBSD</a>, or |
| <a href="http://www.OpenBSD.org/docum.html">OpenBSD</a>. |
| In addition, the FreeBSD |
| <a href="http://www.FreeBSD.org/cgi/man.cgi?query=tuning">tuning</a> |
| manual page contains lots of wisdom, especially regarding sysctl |
| options.</p> |
| |
| <p>Like other OS's, the listen queue is often the <strong>first |
| limit hit</strong>. The following are comments from "Aaron |
| Gifford <agifford@InfoWest.COM>" on how to fix this on |
| BSDI 1.x, 2.x, and FreeBSD 2.0 (and earlier):</p> |
| |
| <p>Edit the following two files:</p> |
| |
| <blockquote> |
| <code>/usr/include/sys/socket.h<br /> |
| /usr/src/sys/sys/socket.h</code> |
| </blockquote> |
| In each file, look for the following: |
| <pre> |
| /* |
| * Maximum queue length specifiable by listen. |
| */ |
| #define SOMAXCONN 5 |
| </pre> |
| Just change the "5" to whatever appears to work. I bumped the |
| two machines I was having problems with up to 32 and haven't |
| noticed the problem since. |
| |
| <p>After the edit, recompile the kernel and recompile the |
| Apache server then reboot.</p> |
| |
| <p>FreeBSD 2.1 seems to be perfectly happy, with SOMAXCONN set |
| to 32 already.</p> |
| |
| <p><a id="detail" name="detail"><strong>Addendum for |
| <em>very</em> heavily loaded BSD servers</strong><br /> |
| </a> from Chuck Murcko <chuck@telebase.com></p> |
| |
| <p>If you're running a really busy BSD Apache server, the |
| following are useful things to do if the system is acting |
| sluggish:</p> |
| |
| <ul> |
| <li>Run vmstat to check memory usage, page/swap rates, |
| <em>etc.</em></li> |
| |
| <li>Run netstat -m to check mbuf usage</li> |
| |
| <li>Run fstat to check file descriptor usage</li> |
| </ul> |
| These utilities give you an idea what you'll need to tune in |
| your kernel, and whether it'll help to buy more RAM. Here are |
| some BSD kernel config parameters (actually BSDI, but pertinent |
| to FreeBSD and other 4.4-lite derivatives) from a system |
| getting heavy usage. The tools mentioned above were used, and |
| the system memory was increased to 48 MB before these tuneups. |
| Other system parameters remained unchanged. |
| <pre> |
| maxusers 256 |
| </pre> |
| Maxusers drives a <em>lot</em> of other kernel parameters: |
| |
| <ul> |
| <li>Maximum # of processes</li> |
| |
| <li>Maximum # of processes per user</li> |
| |
| <li>System wide open files limit</li> |
| |
| <li>Per-process open files limit</li> |
| |
| <li>Maximum # of mbuf clusters</li> |
| |
| <li>Proc/pgrp hash table size</li> |
| </ul> |
| The actual formulae for these derived parameters are in |
| <em>/usr/src/sys/conf/param.c</em>. These calculated parameters |
| can also be overridden (in part) by specifying your own values |
| in the kernel configuration file: |
| <pre> |
| # Network options. NMBCLUSTERS defines the number of mbuf clusters and |
| # defaults to 256. This machine is a server that handles lots of traffic, |
| # so we crank that value. |
| options NMBCLUSTERS=4096 # mbuf clusters at 4096 |
| |
| # |
| # Misc. options |
| # |
| options CHILD_MAX=512 # maximum number of child processes |
| options OPEN_MAX=512 # maximum fds (breaks RPC svcs) |
| </pre> |
| |
| <p>In many cases, NMBCLUSTERS must be set much larger than |
| would appear necessary at first glance. The reason for this is |
| that if the browser disconnects in mid-transfer, the socket fd |
| associated with that particular connection ends up in the |
| TIME_WAIT state for several minutes, during which time its |
| mbufs are not yet freed. Another reason is that, on server |
| timeouts, some connections end up in FIN_WAIT_2 state forever, |
| because this state doesn't time out on the server, and the |
| browser never sent a final FIN. For more details see the <a |
| href="fin_wait_2.html">FIN_WAIT_2</a> page.</p> |
| |
| <p>Some more info on mbuf clusters (from sys/mbuf.h):</p> |
| <pre> |
| /* |
| * Mbufs are of a single size, MSIZE (machine/machparam.h), which |
| * includes overhead. An mbuf may add a single "mbuf cluster" of size |
| * MCLBYTES (also in machine/machparam.h), which has no additional overhead |
| * and is used instead of the internal data area; this is done when |
| * at least MINCLSIZE of data must be stored. |
| */ |
| </pre> |
| |
| <p>CHILD_MAX and OPEN_MAX are set to allow up to 512 child |
| processes (different than the maximum value for processes per |
| user ID) and file descriptors. These values may change for your |
| particular configuration (a higher OPEN_MAX value if you've got |
| modules or CGI scripts opening lots of connections or files). |
| If you've got a lot of other activity besides httpd on the same |
| machine, you'll have to set NPROC higher still. In this |
| example, the NPROC value derived from maxusers proved |
| sufficient for our load.</p> |
| |
| <p>To increase the size of the <code>listen()</code> queue, you |
| need to adjust the value of SOMAXCONN. SOMAXCONN is not derived |
| from maxusers, so you'll always need to increase that yourself. |
| We use a value guaranteed to be larger than Apache's default |
| for the listen() of 128, currently. The actual value for |
| SOMAXCONN is set in <code>sys/socket.h</code>. The best way to |
| adjust this parameter is run-time, rather than changing it in |
| this header file and thus hardcoding a value in the kernel and |
| elsewhere. To do this, edit <code>/etc/rc.local</code> and add |
| the following line:</p> |
| <pre> |
| /usr/sbin/sysctl -w kern.somaxconn=256 |
| </pre> |
| |
| <p>We used <code>256</code> but you can tune it for your own |
| setup. In many cases, however, even the default value of |
| <code>128</code> (for later versions of FreeBSD) is OK.</p> |
| |
| <p><strong>Caveats</strong></p> |
| |
| <p>Be aware that your system may not boot with a kernel that is |
| configured to use more resources than you have available system |
| RAM. <strong>ALWAYS</strong> have a known bootable kernel |
| available when tuning your system this way, and use the system |
| tools beforehand to learn if you need to buy more memory before |
| tuning.</p> |
| |
| <p>RPC services will fail when the value of OPEN_MAX is larger |
| than 256. This is a function of the original implementations of |
| the RPC library, which used a byte value for holding file |
| descriptors. BSDI has partially addressed this limit in its 2.1 |
| release, but a real fix may well await the redesign of RPC |
| itself.</p> |
| |
| <p>Finally, there's the hard limit of child processes |
| configured in Apache.</p> |
| |
| <p>For versions of Apache later than 1.0.5 you'll need to |
| change the definition for <strong>HARD_SERVER_LIMIT</strong> in |
| <em>httpd.h</em> and recompile if you need to run more than the |
| default 150 instances of httpd.</p> |
| |
| <p>From conf/httpd.conf-dist:</p> |
| <pre> |
| # Limit on total number of servers running, <em>i.e.</em>, limit on the number |
| # of clients who can simultaneously connect --- if this limit is ever |
| # reached, clients will be LOCKED OUT, so it should NOT BE SET TOO LOW. |
| # It is intended mainly as a brake to keep a runaway server from taking |
| # Unix with it as it spirals down... |
| |
| MaxClients 150 |
| </pre> |
| Know what you're doing if you bump this value up, and make sure |
| you've done your system monitoring, RAM expansion, and kernel |
| tuning beforehand. Then you're ready to service some serious |
| hits! |
| |
| <p>Thanks to <em>Tony Sanders</em> and <em>Chris Torek</em> at |
| BSDI for their helpful suggestions and information.</p> |
| |
| <p>"M. Teterin" <mi@ALDAN.ziplink.net> writes:</p> |
| |
| <blockquote> |
| It really does help if your kernel and frequently used |
| utilities are fully optimized. Rebuilding the FreeBSD kernel |
| on an AMD-133 (486-class CPU) web-server with<br /> |
| <code>-m486 -fexpensive-optimizations -fomit-frame-pointer |
| -O2</code><br /> |
| helped reduce the number of "unable" errors, because the CPU |
| was often maxed out. |
| </blockquote> |
| |
| <h2><a id="accf" name="accf">Accept filtering on |
| FreeBSD</a></h2> |
| |
| <p>Versions of FreeBSD from August 2000 onwards include a |
| feature called "accept filters" which delay the return from |
| accept() until a condition has been met, e.g. an HTTP request |
| has arrived. This postpones the requirement for a child process |
| to handle the new connection which therefore increases the |
| number of connections that a given number of child processes |
| can handle. It also allows a child process to accomplish more |
| immediately after accept() returns (because the request is |
| already available to be read) so there is less context |
| switching.</p> |
| |
| <p>Accept filters provide the most benefit on servers that are |
| already so busy that they are configured with "<code>KeepAlive |
| Off</code>". <a href="../keepalive.html">HTTP KeepAlive (aka |
| persistent connections)</a> avoids the cost of setting up a new |
| connection for every request, but connections that are being |
| kept alive use up one of the available child processes. Since |
| there is a limited number of child processes this can |
| significantly reduce the capacity of the server. The viewers of |
| a web site will still get a lot of the benefit of persistent |
| connections even with a very small |
| <code>KeepAliveTimeout</code> so you should try reducing it |
| before turning it off altogether.</p> |
| |
| <p>To enable accept filtering, you must either load the |
| appropriate accept filter module, e.g. with the command |
| <code>kldload accf_http</code>, or compile a kernel with |
| <code>options ACCEPT_FILTER_HTTP</code>. Apache will then |
| enable filtering when it is restarted.</p> |
| |
| <p>Accept filters are compiled in if the symbol |
| <code>SO_ACCEPTFILTER</code> is defined on the machine on which |
| Apache is built. Additionally there is a directive <a |
| href="../mod/core.html#acceptfilter">AcceptFilter</a> to switch |
| the filters on or off. The default is on; except when apache is |
| compiled with <code>-D AP_ACCEPTFILTER_ON</code>.</p> |
| |
| <p>See the manual page |
| <a href="http://www.freebsd.org/cgi/man.cgi?query=accf_http">accf_http(9)</a> |
| for more information.</p> |
| |
| <h3>More welcome!</h3> |
| If you have tips to contribute, send mail to <a |
| href="mailto:apache@apache.org">apache@apache.org</a> |
| <!--#include virtual="footer.html" --> |
| </body> |
| </html> |
| |
