Solved

Server PHP script timeout 500 error

Posted on 2014-11-09
9
675 Views
Last Modified: 2014-12-04
Hello,

I have a php script that crawls a bunch of websites for new jobs, curl.

I am getting a timeout for a couple job titles as there are a lot of them on the other websites.

The only function of the server is to find these jobs so I am not concerned about the length of the script running.

I have tried to edit the fcgid module but it still times out around 9 minutes... any where else you suggest or anything you see here?

# This is the Apache server configuration file for providing FastCGI support
# via mod_fcgid
#
# Documentation is available at http://fastcgi.coremail.cn/doc.htm

LoadModule fcgid_module modules/mod_fcgid.so

<IfModule mod_fcgid.c>

<IfModule !mod_fastcgi.c>
    AddHandler fcgid-script fcg fcgi fpl
</IfModule>

  FcgidIPCDir /var/run/mod_fcgid/sock
  FcgidProcessTableFile /var/run/mod_fcgid/fcgid_shm

  FcgidIdleTimeout 40
  FcgidProcessLifeTime 1200
  FcgidMaxProcesses 40
  FcgidMaxProcessesPerClass 8
  FcgidMinProcessesPerClass 0
  FcgidConnectTimeout 1200
  FcgidIOTimeout 1200
  FcgidInitialEnv RAILS_ENV production
  FcgidIdleScanInterval 10

</IfModule>

Open in new window

0
Comment
Question by:movieprodw
  • 4
  • 4
9 Comments
 
LVL 35

Expert Comment

by:Kimputer
ID: 40432234
You should look at php.ini for
max_execution_time (in seconds)
and also apache's httpd.conf parameter TimeOut (in seconds)

Or you should just run the script from commandline instead of web interface
0
 
LVL 109

Accepted Solution

by:
Ray Paseur earned 500 total points
ID: 40432253
A solution that I've found to a similar problem goes something like this:

1. Before each iteration in the long-running process, perhaps before you use cURL to acquire the HTML documents from the other sites, use set_time_limit(XX) with a value in XX that makes sense for the duration of that step of the process.  It might be 5 or 15 seconds - whatever you need to get done with that step.  Don't use 0 since an errant script loop will never be aborted.

2. At the same time, produce some amount of browser output.  You may be able to just echo the name of the URL, or something like that.  Then use flush() to cause PHP to poke the web server.  You may find that you need to pad the output in order to make it long enough that it gets past PHP and server buffering and actually gets sent to the browser.

Taken together these two things will (1) keep the server from killing the PHP script prematurely and (2) keep the server from thinking the PHP script has gone to sleep and (3) give you a browser-visual indicator of the process.  You may want to read the man pages for flush() carefully; it's not 100% intuitive.

Here's a simulation script that seems to work well for me:
http://iconoun.com/demo/force_flush.php

<?php // force_flush.php
error_reporting(E_ALL);
date_default_timezone_set('America/New_York');


// SIMULATE A LONG-RUNNING JOB WITH INTERMITTENT BROWSER OUTPUT
// FORCE FLUSH THE BROWSER OUTPUT BEFORE THE END OF THE PAGE
// PAD THE OUTPUT STRING TO MAKE IT LONG SO THE SERVER SENDS IT


function my_echo($str, $len=512, $pad=' ')
{
    // PAD THE STRING WITH INVISIBLE WHITESPACE OR SIMILAR
    $str = str_pad($str, strlen($str) + $len, $pad);
    echo $str;
    echo PHP_EOL;
    flush();
}


// TEST THE URL ARGUMENT FOR "b=y" AND OPTIONALLY START THE OUTPUT BUFFER
$b = (!empty($_GET["b"])) ? $_GET["b"] : NULL;
if ($b == 'y') ob_start();


// RUN THE TEST MESSAGES, WAITING BETWEEN THE MESSAGES
my_echo("<br/>Hello #1 " . date('c'));
sleep(1);
my_echo("<br/>Hello #2 " . date('c'));
sleep(2);
my_echo("<br/>Hello #3 " . date('c'));
sleep(3);
my_echo("<br/>Hello #4 " . date('c'));
sleep(4);
my_echo("<br/>Hello #5 " . date('c'));
sleep(1);
my_echo("<br/>Hello #6 " . date('c'));

Open in new window

0
 
LVL 1

Author Comment

by:movieprodw
ID: 40441009
Hello,

Okay, I will try this out, it is a bummer you can not just set a timeout, it times out at 7 minutes ~420 seconds no matter what I set the max at.

Matt
0
Enterprise Mobility and BYOD For Dummies

Like “For Dummies” books, you can read this in whatever order you choose and learn about mobility and BYOD; and how to put a competitive mobile infrastructure in place. Developed for SMBs and large enterprises alike, you will find helpful use cases, planning, and implementation.

 
LVL 1

Author Comment

by:movieprodw
ID: 40441013
You can see the settings here, I set the max huge for testing

http://makemodelsearch.com/trash2.php
0
 
LVL 35

Expert Comment

by:Kimputer
ID: 40442177
You still didn't mention the httpd.conf Timeout setting.
0
 
LVL 1

Author Comment

by:movieprodw
ID: 40442940
#
# Timeout: The number of seconds before receives and sends time out.
#
Timeout 60
0
 
LVL 35

Expert Comment

by:Kimputer
ID: 40445021
Can you try Timeout 600 (restart httpd) and see if it gets better?
0
 
LVL 1

Author Closing Comment

by:movieprodw
ID: 40480612
Worked, thanks
0
 
LVL 35

Expert Comment

by:Kimputer
ID: 40480617
What was the solution then?
0

Featured Post

Back Up Your Microsoft Windows Server®

Back up all your Microsoft Windows Server – on-premises, in remote locations, in private and hybrid clouds. Your entire Windows Server will be backed up in one easy step with patented, block-level disk imaging. We achieve RTOs (recovery time objectives) as low as 15 seconds.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Password hashing is better than message digests or encryption, and you should be using it instead of message digests or encryption.  Find out why and how in this article, which supplements the original article on PHP Client Registration, Login, Logo…
Fine Tune your automatic Updates for Ubuntu / Debian
Explain concepts important to validation of email addresses with regular expressions. Applies to most languages/tools that uses regular expressions. Consider email address RFCs: Look at HTML5 form input element (with type=email) regex pattern: T…
Get a first impression of how PRTG looks and learn how it works.   This video is a short introduction to PRTG, as an initial overview or as a quick start for new PRTG users.

803 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question