[vox-tech] php-cgi hanging on semop()

[Kyle Ambroff]

On Wed, Nov 24, 2010 at 4:59 PM, Bill Kendrick <nbs at sonic.net> wrote:
>  Process 17180 attached - interrupt to quit
>  semop(229383, {{0, -1, SEM_UNDO}}, 1^C <unfinished ...>
>  Process 17180 detached

If I were looking into this, I would probably use gdb to get a better
understanding of the state of the php5-cgi process and then dig into
the source code. I'm guessing you either started using a new extension
module, or installed an update recently that introduced some resource
exhaustion bug that causes deadlock.

I think the output of this would be pretty helpful:

 sudo gdb --pid=$(pidof php5-cgi) -ex 'thread apply all bt full' -batch

If you can install debug symbols that would be helpful too. If you are
using debian php5-dbg should help.

If the backtrace is mostly in php code, you can probably tease a good
trace out of gdb. If you were using Python I would give you the answer
right now :), but I'm not that familiar with the inner workings of the
php VM.

The output of lsof -p $(pidof php5-cgi) might be nice too, since some
of the threads might show that some of your code is hanging on some
IO. I've run into a similar situation that ended up being one of our
custom apache modules that had exhausted it's pool of memcached
connections due to a leak, and the combination of lsof and gdb made
the problem really easy to spot.

-Kyle