Your browser (Internet Explorer 6) is out of date. It has known security flaws and may not display all features of this and other websites. Learn how to update your browser.
X
Post

Resolving php-fpm nginx 504 Gateway-Timeouts

Scotch Tape Icon by Arnoud Warschauer
Scotch Tape Icon by Arnoud Warschauer

I ran into this issue the other day on an Ubuntu 11.04 server running WordPress. The solutions I found searching online weren’t all that helpful so I wanted to share the steps I followed to sort out the issue.

Check Your nginx and php-fpm Log Files

Its always a good idea to begin with the log files and start working from any errors they reveal.

nginx error log

sudo cat /var/log/nginx/error.log

php-fpm log

sudo cat /var/log/php5-fpm.log

In the particular case I was troubleshooting neither log file had any clues that could help me discover the root cause at this point. Your mileage may vary.

Tweak Your Timeout Settings

I came across a helpful clue online about changing the PHP max_execution_time setting in your php.ini file. In the case of php-fpm setups there’s also a request_terminate_timeout setting in your php-fpm config file.

According to the php-fpm wiki request_terminate_timeout is useful in cases where the max_execution_time setting in php.ini isn’t enforced. The default value for request_terminate_timeout is 0 which means it doesn’t get used unless you set it to a proper value.

Setting values in both places will make sure the gateway (php-fpm) and your actual script both timeout after a reasonable interval. To help with troubleshooting I set them both to 30 seconds which is the default for max_execution_time.

make sure max_execution_time is set to 30 seconds in your php.ini file

max_execution_time = 30

set request_terminate_timeout to 30 seconds in your www.conf file

request_terminate_timeout = 30s

With those new settings in place be sure to restart nginx and php-fpm. Now when you hit the page causing the 504 error you should start to see request timeout messages in your php-fpm log file.

Setup The php-fpm Slow Log

Once your timeout settings are in place the problem is effectively isolated to some runaway code that only stops when it hits one of the timeouts. The next step is identifying where the runaway code is. That’s were the php-fpm slow log comes into play.

By configuring the request_slowlog_timeout and setting up a slowlog file, php-fpm will dump a php backtrace into to the slow log file whenever a script hits the timeout value.

set the request_slowlog_timeout to 25 seconds in your www.conf

request_slowlog_timeout = 25s

set a location for the slowlog file in www.conf

slowlog = /var/log/php5-fpm.slow.log

Now when you restart everything and hit the page causing the error you’ll get a really helpful backtrace in your slow log file. The backtrace will show you the specific line of PHP code that was being executed when it hit the 25 second limit.

Here’s an example entry from the server I was troubleshooting.

script_filename = /home/apps/hublistapp.com/public/wp-admin/post.php
[0x0000000002d15150] file_get_contents() /home/apps/hublistapp.com/public/wp-content/themes/fastblog/functions.php:349
[0x00007fff0ff0e300] fastblog_save_post() unknown:0
[0x0000000002d14500] call_user_func_array() /home/apps/hublistapp.com/public/wp-includes/plugin.php:395
[0x0000000002d11940] do_action() /home/apps/hublistapp.com/public/wp-includes/post.php:2634
[0x0000000002d111a0] wp_insert_post() /home/apps/hublistapp.com/public/wp-includes/post.php:2689
[0x0000000002d0fa20] wp_update_post() /home/apps/hublistapp.com/public/wp-admin/includes/post.php:225
[0x0000000002d0d0f8] edit_post() /home/apps/hublistapp.com/public/wp-admin/post.php:207

Reading this entry reveals the problem was with a call to file_get_contents() on line 349 of the functions.php file. In this particular case file_get_contents was being called with an error control operator @ as part of some URL shortening routine that the theme was trying to run whenever a new blog post was created.

For whatever reason that call would run and never timeout out until it hit the execution limits which ended up displaying the 504 timeout error. I was able turn off the URL shortening feature from the theme options panel which skipped the block of code causing problems.

Hopefully these steps help others troubleshoot similar issues. If you have additional tips share them in the comments!

  • guest

    I have image hosting site, both max_execution_time and request_terminate_timeout is set to 240s, yet the script times out after 60 seconds :/
    I have no clue what else could cause it.

  • http://rawberg.com/blog David Feinberg

    is there any code in your scripts using the error control operator? is anything making a call to an external service? is PHP error logging turned on? is anything showing up in the error logs?

  • guest

    It’s Mihalism Multihost script, haven’t really poked around source code all that much. There is no errors in php log. Time out occurs during remote upload. Actually the php script continues to run in the background even after nginx shows the timeout message. Seems to be an issue of nginx then.

  • http://rawberg.com/blog David Feinberg

    I’m not familiar with that script. Your best bet is to post your issue on their user forums and see if anyone else has run into a similar issue using it with Nginx setups.

  • guest

    Solved it. As expected, problem was nginx config. I added 3 following values and it works like a charm now:
    client_header_timeout 240;
    client_body_timeout 240;
    fastcgi_read_timeout 240;

  • http://rawberg.com/blog David Feinberg

    That’s great to hear! Thanks for keeping us posted and sharing your solution!

  • Nicolas Embleton

    Interesting post. Thanks. I also believe that the NGinx conf is the one that everybody ( including me ) forget.

  • Guest

    This just really helped me, where request_terminate_timeout threw an error when checking the config.

    I haven’t looked into it, but could request_terminate_timeout be deprecated? I’m running nginx 0.7.65 on Ubuntu 10.04 LTS.

  • http://rawberg.com/blog David Feinberg

    Glad to hear it helped you out! Regarding request_terminate_timeout it’s actually a php-fpm setting. Seems like you may have been trying to put it in the wrong place (doh).

  • Carole

    how do i turn off gateway

  • http://rawberg.com/blog David Feinberg

    could you elaborate a bit more, what exactly are you trying to do?

  • http://twitter.com/javipas JaviPas

    The PHP-FPM slow log saved my life a couple of days ago, I didn’t know about anything about that one and has proved really useful to detect why one of my websites was loading so slowly.

    I can’t really say how much thankful I am for your post. Great job, congratulations!!

  • http://rawberg.com/blog David Feinberg

    So glad it was helpful for you, thanks for the positive feedback!

  • http://twitter.com/vjom Ville Mattila

    I collected some more configuration to my blog post http://www.villescorner.com/2012/02/nginx-php-fpm-file-upload-battle.html – especially the maximum body size limits are also affecting the nginx & php-fpm timeouts in some cases.