We are being overwhelmed with httpd requests that never seem to quit. Most of them seem to be coming from various bots that search engines send out. Here’s an example:
$ netstat -tpu|grep http
tcp 1 0 hep.uchicago.edu:www msnbot.msn.com:20281 CLOSE_WAIT 5371/httpd
tcp 1 0 hep.uchicago.edu:www msnbot.msn.com:11460 CLOSE_WAIT 5345/httpd
tcp 1 0 hep.uchicago.edu:www msnbot.msn.com:55572 CLOSE_WAIT 5341/httpd
tcp 1 0 hep.uchicago.edu:www crawl-66-249-72-201.g:56629 CLOSE_WAIT 6151/httpd
tcp 1 0 hep.uchicago.edu:www crawl-66-249-72-201.g:50213 CLOSE_WAIT 6778/httpd
tcp 1 0 hep.uchicago.edu:www crawl-66-249-72-201.g:47647 CLOSE_WAIT 7473/httpd
tcp 1 0 hep.uchicago.edu:www crawl-66-249-72-201.g:61674 CLOSE_WAIT 7166/httpd
tcp 1 0 hep.uchicago.edu:www crawl-66-249-72-201.g:43990 CLOSE_WAIT 5806/httpd
tcp 1 0 hep.uchicago.edu:www crawl-66-249-72-201.g:40917 CLOSE_WAIT 5430/httpd
tcp 0 0 hep.uchicago.edu:www crawl-66-249-72-201.g:37338 ESTABLISHED 7058/httpd
tcp 1 0 hep.uchicago.edu:www msnbot.msn.com:49380 CLOSE_WAIT 7263/httpd
tcp 1 0 hep.uchicago.edu:www crawl-66-249-72-201.g:32959 CLOSE_WAIT 7064/httpd
tcp 1 0 hep.uchicago.edu:www crawl-66-249-72-201.g:38831 CLOSE_WAIT 5343/httpd
tcp 1 0 hep.uchicago.edu:www crawl-66-249-72-201.g:61835 CLOSE_WAIT 5344/httpd
After a couple of hours, our webserver is inaccessible due to the large number of daemons running. The solution, automatically restart httpd every hour. Added the following to the crontab:
# Each hour, restart the httpd2 daemon to clear them out
0 * * * * /etc/rc.d/init.d/httpd2 restart 2 >& /dev/null