| <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN"> |
| <html> |
| <head> |
| <title>Running a High-Performance Web Server for BSD</title> |
| </head> |
| |
| <!-- Background white, links blue (unvisited), navy (visited), red (active) --> |
| <BODY |
| BGCOLOR="#FFFFFF" |
| TEXT="#000000" |
| LINK="#0000FF" |
| VLINK="#000080" |
| ALINK="#FF0000" |
| > |
| <A NAME="initial"> |
| <!--#include virtual="header.html" --> |
| </A> |
| <H1 ALIGN="CENTER">Running a High-Performance Web Server for BSD</H1> |
| |
| Like other OS's, the listen queue is often the <b>first limit hit</b>. The |
| following are comments from "Aaron Gifford <agifford@InfoWest.COM>" |
| on how to fix this on BSDI 1.x, 2.x, and FreeBSD 2.0 (and earlier): |
| |
| <p> |
| |
| Edit the following two files: |
| <blockquote><code> /usr/include/sys/socket.h <br> |
| /usr/src/sys/sys/socket.h </code></blockquote> |
| In each file, look for the following: |
| <pre> |
| /* |
| * Maximum queue length specifiable by listen. |
| */ |
| #define SOMAXCONN 5 |
| </pre> |
| |
| Just change the "5" to whatever appears to work. I bumped the two |
| machines I was having problems with up to 32 and haven't noticed the |
| problem since. |
| |
| <p> |
| |
| After the edit, recompile the kernel and recompile the Apache server |
| then reboot. |
| |
| <P> |
| |
| FreeBSD 2.1 seems to be perfectly happy, with SOMAXCONN |
| set to 32 already. |
| |
| <p> |
| |
| <A NAME="detail"> |
| <b>Addendum for <i>very</i> heavily loaded BSD servers</b><br> |
| </A> |
| from Chuck Murcko <chuck@telebase.com> |
| |
| <p> |
| |
| If you're running a really busy BSD Apache server, the following are useful |
| things to do if the system is acting sluggish:<p> |
| |
| <ul> |
| |
| <li> Run vmstat to check memory usage, page/swap rates, etc. |
| |
| <li> Run netstat -m to check mbuf usage |
| |
| <li> Run fstat to check file descriptor usage |
| |
| </ul> |
| |
| These utilities give you an idea what you'll need to tune in your kernel, |
| and whether it'll help to buy more RAM. |
| |
| Here are some BSD kernel config parameters (actually BSDI, but pertinent to |
| FreeBSD and other 4.4-lite derivatives) from a system getting heavy usage. |
| The tools mentioned above were used, and the system memory was increased to |
| 48 MB before these tuneups. Other system parameters remained unchanged. |
| |
| <p> |
| |
| <pre> |
| maxusers 256 |
| </pre> |
| |
| Maxusers drives a <i>lot</i> of other kernel parameters: |
| |
| <ul> |
| |
| <li> Maximum # of processes |
| |
| <li> Maximum # of processes per user |
| |
| <li> System wide open files limit |
| |
| <li> Per-process open files limit |
| |
| <li> Maximum # of mbuf clusters |
| |
| <li> Proc/pgrp hash table size |
| |
| </ul> |
| |
| The actual formulae for these derived parameters are in |
| <i>/usr/src/sys/conf/param.c</i>. |
| These calculated parameters can also be overridden (in part) by specifying |
| your own values in the kernel configuration file: |
| |
| <pre> |
| # Network options. NMBCLUSTERS defines the number of mbuf clusters and |
| # defaults to 256. This machine is a server that handles lots of traffic, |
| # so we crank that value. |
| options SOMAXCONN=256 # max pending connects |
| options NMBCLUSTERS=4096 # mbuf clusters at 4096 |
| |
| # |
| # Misc. options |
| # |
| options CHILD_MAX=512 # maximum number of child processes |
| options OPEN_MAX=512 # maximum fds (breaks RPC svcs) |
| </pre> |
| |
| SOMAXCONN is not derived from maxusers, so you'll always need to increase |
| that yourself. We used a value guaranteed to be larger than Apache's |
| default for the listen() of 128, currently. |
| |
| <p> |
| |
| In many cases, NMBCLUSTERS must be set much larger than would appear |
| necessary at first glance. The reason for this is that if the browser |
| disconnects in mid-transfer, the socket fd associated with that particular |
| connection ends up in the TIME_WAIT state for several minutes, during |
| which time its mbufs are not yet freed. Another reason is that, on server |
| timeouts, some connections end up in FIN_WAIT_2 state forever, because |
| this state doesn't time out on the server, and the browser never sent |
| a final FIN. For more details see the |
| <A HREF="fin_wait_2.html">FIN_WAIT_2</A> page. |
| |
| <p> |
| |
| Some more info on mbuf clusters (from sys/mbuf.h): |
| <pre> |
| /* |
| * Mbufs are of a single size, MSIZE (machine/machparam.h), which |
| * includes overhead. An mbuf may add a single "mbuf cluster" of size |
| * MCLBYTES (also in machine/machparam.h), which has no additional overhead |
| * and is used instead of the internal data area; this is done when |
| * at least MINCLSIZE of data must be stored. |
| */ |
| </pre> |
| |
| <p> |
| |
| CHILD_MAX and OPEN_MAX are set to allow up to 512 child processes (different |
| than the maximum value for processes per user ID) and file descriptors. |
| These values may change for your particular configuration (a higher OPEN_MAX |
| value if you've got modules or CGI scripts opening lots of connections or |
| files). If you've got a lot of other activity besides httpd on the same |
| machine, you'll have to set NPROC higher still. In this example, the NPROC |
| value derived from maxusers proved sufficient for our load. |
| |
| <p> |
| |
| <b>Caveats</b> |
| |
| <p> |
| |
| Be aware that your system may not boot with a kernel that is configured |
| to use more resources than you have available system RAM. <b>ALWAYS</b> |
| have a known bootable kernel available when tuning your system this way, |
| and use the system tools beforehand to learn if you need to buy more |
| memory before tuning. |
| |
| <p> |
| |
| RPC services will fail when the value of OPEN_MAX is larger than 256. |
| This is a function of the original implementations of the RPC library, |
| which used a byte value for holding file descriptors. BSDI has partially |
| addressed this limit in its 2.1 release, but a real fix may well await |
| the redesign of RPC itself. |
| |
| <p> |
| |
| Finally, there's the hard limit of child processes configured in Apache. |
| |
| <p> |
| |
| For versions of Apache later than 1.0.5 you'll need to change the |
| definition for <b>HARD_SERVER_LIMIT</b> in <i>httpd.h</i> and recompile |
| if you need to run more than the default 150 instances of httpd. |
| |
| <p> |
| |
| From conf/httpd.conf-dist: |
| |
| <pre> |
| # Limit on total number of servers running, i.e., limit on the number |
| # of clients who can simultaneously connect --- if this limit is ever |
| # reached, clients will be LOCKED OUT, so it should NOT BE SET TOO LOW. |
| # It is intended mainly as a brake to keep a runaway server from taking |
| # Unix with it as it spirals down... |
| |
| MaxClients 150 |
| </pre> |
| |
| Know what you're doing if you bump this value up, and make sure you've |
| done your system monitoring, RAM expansion, and kernel tuning beforehand. |
| Then you're ready to service some serious hits! |
| |
| <p> |
| |
| Thanks to <i>Tony Sanders</i> and <i>Chris Torek</i> at BSDI for their |
| helpful suggestions and information. |
| |
| <P> |
| |
| "M. Teterin" <mi@ALDAN.ziplink.net> writes:<P> |
| <blockquote>It really does help if your kernel and frequently used utilities |
| are fully optimized. Rebuilding the FreeBSD kernel on an AMD-133 |
| (486-class CPU) web-server with<BR> |
| <code> -m486 -fexpensive-optimizations -fomit-frame-pointer -O2</code><BR> |
| helped reduce the number of "unable" errors, because the CPU was |
| often maxed out.</blockquote> |
| <P> |
| |
| <HR> |
| |
| <H3>More welcome!</H3> |
| |
| If you have tips to contribute, send mail to <a |
| href="mailto:brian@organic.com">brian@organic.com</a> |
| |
| <!--#include virtual="footer.html" --> |
| </body></html> |
| |