Ole Morten Halvorsen

March 6, 2008
Fixing Apache Segmentation Faults Caused by PHP

Ever seen this in your Apache error log?

[notice] child pid 10024 exit signal Segmentation fault (11)

If you have, read on. At my day job I deal with both Apache and PHP a lot. If you have ever tried to figure out why your PHP code seems to cause Apache segmentation faults, you probably experienced the same as I did. Lots of pain, frustration and headaches. However all is not lost. There is a way to figure out what PHP code is making Apache act crazy. I’ll try to explain how to debug Apache using gdb to locate that nasty bug that is causing you to lose your precious beauty sleep.

Before we begin

Now, Apache does not dump core by default. We need to do some work before that happens. If you don’t like to get your hands dirty compiling stuff manually, you should leave now, this is not for you.

Still here? Good! Before we start, make sure you have:

  1. Root access to your web server
  2. Apache source code and everything needed to (re-)compile Apache
  3. PHP source code (or just download the .gbdinit file)

Make Apache dump core

The first step is to make Apache not change user when it starts up and forks, so we will make it run as root the whole time. To accomplish this we need to compile Apache with -DBIG_SECURITY_HOLE. For obvious reasons, this is not recommended for production.

make clean
./configure && make && make install

Now specify the “root” as the “User” in httpd.conf. While we’re editing httpd.conf we’ll add a setting to specify where Apache should put our core dump.

User nobody
CoreDumpDirectory /tmp/apache

Make sure /tmp/apache exists

mkdir -p /tmp/apache

On most systems the core file size is set to zero by default. We’ll go ahead and change it to unlimited.

ulimit -c 0

You can check your current core dump file size limit by running

ulimit -a

Restart Apache

apachectl restart

Next time Apache crashes with a Segmentation fault it should make us a core dump. If you did everything correctly you should see something like this in the apache error_log.

[notice] child pid 16430 exit signal Segmentation fault (11),  
possible coredump in /tmp/apache

Note: For reasons unclear to me my dump file ended up on the root (/) and not in the directory we specified with CoreDumpDirectory. If anyone knows why please drop me a comment.

Making sense of the core dump

At this point we could run gdb and get a backtrace, however that will only show us the function called inside php itself and not pinpoint where in our php code the problem is. To get a backtrace of our PHP code we need to use the “dump_bt” function inside the .gbdinit file. Copy the .gbdinit file to your home directory

cp <php_source>/.gdbinit ~/

Start gdb with the path to httpd from the source as first parameter and the path to the core dump file as second

cd ~
gdb <apache_source>/src/httpd /tmp/apache/<core_file>

Run bt_dump from .gdbinit:

(gdb) dump_bt executor_globals.current_execute_data

Instead of a internal PHP backtrace we should get a nice backtrace of our PHP code.

[0xbf75f6fc] query() /home/site/www/lib/ezdb/classes/ezmysqldb.php:767
[0xbf76249c] unlock() /home/site/www/lib/ezdb/classes/ezmysqldb.php:484
[0xbf7627bc] query() /home/site/www/lib/ezdb/classes/ezmysqldb.php:767
[0xbf76555c] unlock() /home/site/www/lib/ezdb/classes/ezmysqldb.php:484
[0xbf76587c] query() /home/site/www/lib/ezdb/classes/ezmysqldb.php:767
[0xbf76861c] unlock() /home/site/www/lib/ezdb/classes/ezmysqldb.php:484
[0xbf76893c] query() /home/site/www/lib/ezdb/classes/ezmysqldb.php:767
[0xbf76b6dc] unlock() /home/site/www/lib/ezdb/classes/ezmysqldb.php:484
[0xbf76b9fc] query() /home/site/www/lib/ezdb/classes/ezmysqldb.php:767
[0xbf76e79c] unlock() /home/site/www/lib/ezdb/classes/ezmysqldb.php:484
[0xbf76eabc] query() /home/site/www/lib/ezdb/classes/ezmysqldb.php:767
[0xbf77185c] unlock() /home/site/www/lib/ezdb/classes/ezmysqldb.php:484
[0xbf771b7c] query() /home/site/www/lib/ezdb/classes/ezmysqldb.php:767

In this example we can clearly see there is a recursive problem with query() and unlock() on line 767 and 484 in ezmysqldb.php. With the help of this backtrace in only took a quick look in ezmysqldb.php before I had a bug report written and posted: http://issues.ez.no/11038.

Thanks to my good friend and co-worker Derick Rethans for telling me about the “dump_bt” function.


#1 Derick

Posted: March 7, 2008

A segfault often shows in a blank page as well. If something like infinite recursion happens, and you can reproduce it on a development / non-production machine, another (much easier) way to figure out why things mess up is by using Xdebug. It features a infinite recursion check that shows you the stack trace as part of the scripts output; see http://www.xdebug.org/docs/basic#max_nesting_level and http://www.xdebug.org/docs/stack_trace

#2 Ole Morten

Posted: March 9, 2008

Yes, Xdebug should really be your first line of defense. Loading Apache into gdb should be your last line of defense.

Leave a comment

Commenting has been disabled. This entry was posted more than 2 weeks ago.

Created by Ole Morten Halvorsen.   Powered by Django.