[vox-tech] php-cgi hanging on semop()

[Bill Kendrick]

So I'm noticing some stability issues with a website.

* lighttpd 1.4.x
* php 5.3.x
* mysql 5.1.x

The symptoms to the outside world include lagging connections and,
eventually, Error 500s from the webserver.

Internally, I run "mytop" (basically, a 'top'-like interface that
parses the results of a "show full processlist" query).  mytop shows
me what connections are active.

Typically, most of our connections are relatively active (only idle a
couple of seconds, at most; If the website's not very busy, the idle
time in some of them can climb in idleness).

However, in the situation I'm seeing, all of the connections from the
webserver are idle, and their idle times continue to increase.
Basically, the current connections between the website and the
database aren't doing anything (not running queries for the website),
and are just sitting there exhausting MySQL's connection pool.

Since this has only started happening recently, and usually after-hours,
my quick-fix has been to do this:

  sudo /etc/init.d/lighttpd stop   # stop webserver
  sudo killall php-cgi             # kill all php-cgi processes
  sudo killall -2 php-cgi          # no, REALLY kill them
  sudo /etc/init.d/lighttpd start  # restart webserver

And things have typically been fine for at least another day.


Today, it has happend a _FEW_ times during business hours, so I
get to drop everything and investigate.  In my job, I am a
literal Jack of All Trades, Master of None.  (On a good day,
I spend my time doing web app development, so I suppose that's
my "main" job ;) )

Google hasn't helped me much, sadly, so I've decided to post here :)


Briefly, here's what I get if I try to attach an strace to
one of the running php-cgi processes.  Note: I see the
same thing:

  - before stopping lighttpd
  - after stopping lighttpd, but before 'killall php-cgi'
  - after 'killall php-cgi', but (obviously), before 'killall -2...'

So here's the clue, gang.  Like, ZOINKS!

  Process 17180 attached - interrupt to quit
  semop(229383, {{0, -1, SEM_UNDO}}, 1^C <unfinished ...>
  Process 17180 detached                                 


In my investigation, I've learned that "ipcs" might be my
friend.  Sadly(?), the server's been chugging along fine
since the last restart.  In the meantime, in case it helps,
here are some logs from /var/log/lighttpd/error.log

[crash! restart]

  2010-11-24 15:36:07: (log.c.166) server started
  2010-11-24 15:38:00: (mod_fastcgi.c.2582) unexpected end-of-file (perhaps the fastcgi process died): pid: 17185 socket: unix:/tmp/php.socket-2
  2010-11-24 15:38:00: (mod_fastcgi.c.3382) response already sent out, but backend returned error on socket: unix:/tmp/php.socket-2 for /index.php?, terminating connection
  2010-11-24 15:45:19: (mod_fastcgi.c.2582) unexpected end-of-file (perhaps the fastcgi process died): pid: 17170 socket: unix:/tmp/php.socket-0
  2010-11-24 15:45:19: (mod_fastcgi.c.3367) response not received, request sent: 991 on socket: unix:/tmp/php.socket-0 for /index.php?, closing connection
  2010-11-24 15:49:11: (mod_fastcgi.c.2582) unexpected end-of-file (perhaps the fastcgi process died): pid: 17178 socket: unix:/tmp/php.socket-1
  2010-11-24 15:49:11: (mod_fastcgi.c.3367) response not received, request sent: 851 on socket: unix:/tmp/php.socket-1 for /index.php?, closing connection
  2010-11-24 15:52:32: (mod_fastcgi.c.2582) unexpected end-of-file (perhaps the fastcgi process died): pid: 17178 socket: unix:/tmp/php.socket-1
  2010-11-24 15:52:32: (mod_fastcgi.c.3367) response not received, request sent: 923 on socket: unix:/tmp/php.socket-1 for /index.php?, closing connection
  2010-11-24 15:53:33: (server.c.1503) server stopped by UID = 0 PID = 21022

[crash! restart]

  2010-11-24 15:53:48: (log.c.166) server started
  2010-11-24 15:59:16: (mod_fastcgi.c.3011) backend is overloaded; we'll disable it for 1 seconds and send the request to another backend instead: reconnects: 0 load: 537
  2010-11-24 15:59:16: (mod_fastcgi.c.3011) backend is overloaded; we'll disable it for 1 seconds and send the request to another backend instead: reconnects: 1 load: 537
  2010-11-24 15:59:16: (mod_fastcgi.c.3011) backend is overloaded; we'll disable it for 1 seconds and send the request to another backend instead: reconnects: 2 load: 537
  2010-11-24 15:59:16: (mod_fastcgi.c.3011) backend is overloaded; we'll disable it for 1 seconds and send the request to another backend instead: reconnects: 3 load: 537
  2010-11-24 15:59:16: (mod_fastcgi.c.3608) all handlers for /index.php? on .php are down.
  2010-11-24 15:59:18: (mod_fastcgi.c.2774) fcgi-server re-enabled:  0 /tmp/php.socket
  2010-11-24 15:59:18: (mod_fastcgi.c.2774) fcgi-server re-enabled:  0 /tmp/php.socket
  2010-11-24 15:59:18: (mod_fastcgi.c.2774) fcgi-server re-enabled:  0 /tmp/php.socket
  2010-11-24 15:59:18: (mod_fastcgi.c.2774) fcgi-server re-enabled:  0 /tmp/php.socket

["backend is overloaded", "all handlers... are down", "fcgi-server re-enabled"
 repeats a bunch of times...]

  2010-11-24 16:01:37: (mod_fastcgi.c.3608) all handlers for /index.php? on .php are down.
  2010-11-24 16:01:37: (server.c.1503) server stopped by UID = 0 PID = 23434
  2010-11-24 16:01:37: (log.c.166) server started
  2010-11-24 16:07:26: (mod_fastcgi.c.2582) unexpected end-of-file (perhaps the fastcgi process died): pid: 23459 socket: unix:/tmp/php.socket-0
  2010-11-24 16:07:26: (mod_fastcgi.c.3367) response not received, request sent: 1091 on socket: unix:/tmp/php.socket-0 for /index.php?, closing connection
  2010-11-24 16:08:29: (connections.c.294) SSL: 1 error:1407609C:SSL routines:SSL23_GET_CLIENT_HELLO:http request

[so far, followed by some repearts of the "mod_fastcgi" errors...]


Any ideas?  Suggestions for next steps to determine the
cause of the issue?  In the meantime, I'm off to learn
about "ipcs" so that I'm prepared to weild it as a weapon
the next time this happens. :)

PS - Bonus hint, the closest thing I've seen to a similar
issue that somehow had seemed to have to do with SSL mutex cache.

Thanks in advance!

-- 
-bill!
Sent from my computer