?
Solved

Server PHP script timeout 500 error

Posted on 2014-11-09
9
Medium Priority
?
877 Views
Last Modified: 2014-12-04
Hello,

I have a php script that crawls a bunch of websites for new jobs, curl.

I am getting a timeout for a couple job titles as there are a lot of them on the other websites.

The only function of the server is to find these jobs so I am not concerned about the length of the script running.

I have tried to edit the fcgid module but it still times out around 9 minutes... any where else you suggest or anything you see here?

# This is the Apache server configuration file for providing FastCGI support
# via mod_fcgid
#
# Documentation is available at http://fastcgi.coremail.cn/doc.htm

LoadModule fcgid_module modules/mod_fcgid.so

<IfModule mod_fcgid.c>

<IfModule !mod_fastcgi.c>
    AddHandler fcgid-script fcg fcgi fpl
</IfModule>

  FcgidIPCDir /var/run/mod_fcgid/sock
  FcgidProcessTableFile /var/run/mod_fcgid/fcgid_shm

  FcgidIdleTimeout 40
  FcgidProcessLifeTime 1200
  FcgidMaxProcesses 40
  FcgidMaxProcessesPerClass 8
  FcgidMinProcessesPerClass 0
  FcgidConnectTimeout 1200
  FcgidIOTimeout 1200
  FcgidInitialEnv RAILS_ENV production
  FcgidIdleScanInterval 10

</IfModule>

Open in new window

0
Comment
Question by:movieprodw
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 4
  • 4
9 Comments
 
LVL 36

Expert Comment

by:Kimputer
ID: 40432234
You should look at php.ini for
max_execution_time (in seconds)
and also apache's httpd.conf parameter TimeOut (in seconds)

Or you should just run the script from commandline instead of web interface
0
 
LVL 111

Accepted Solution

by:
Ray Paseur earned 2000 total points
ID: 40432253
A solution that I've found to a similar problem goes something like this:

1. Before each iteration in the long-running process, perhaps before you use cURL to acquire the HTML documents from the other sites, use set_time_limit(XX) with a value in XX that makes sense for the duration of that step of the process.  It might be 5 or 15 seconds - whatever you need to get done with that step.  Don't use 0 since an errant script loop will never be aborted.

2. At the same time, produce some amount of browser output.  You may be able to just echo the name of the URL, or something like that.  Then use flush() to cause PHP to poke the web server.  You may find that you need to pad the output in order to make it long enough that it gets past PHP and server buffering and actually gets sent to the browser.

Taken together these two things will (1) keep the server from killing the PHP script prematurely and (2) keep the server from thinking the PHP script has gone to sleep and (3) give you a browser-visual indicator of the process.  You may want to read the man pages for flush() carefully; it's not 100% intuitive.

Here's a simulation script that seems to work well for me:
http://iconoun.com/demo/force_flush.php

<?php // force_flush.php
error_reporting(E_ALL);
date_default_timezone_set('America/New_York');


// SIMULATE A LONG-RUNNING JOB WITH INTERMITTENT BROWSER OUTPUT
// FORCE FLUSH THE BROWSER OUTPUT BEFORE THE END OF THE PAGE
// PAD THE OUTPUT STRING TO MAKE IT LONG SO THE SERVER SENDS IT


function my_echo($str, $len=512, $pad=' ')
{
    // PAD THE STRING WITH INVISIBLE WHITESPACE OR SIMILAR
    $str = str_pad($str, strlen($str) + $len, $pad);
    echo $str;
    echo PHP_EOL;
    flush();
}


// TEST THE URL ARGUMENT FOR "b=y" AND OPTIONALLY START THE OUTPUT BUFFER
$b = (!empty($_GET["b"])) ? $_GET["b"] : NULL;
if ($b == 'y') ob_start();


// RUN THE TEST MESSAGES, WAITING BETWEEN THE MESSAGES
my_echo("<br/>Hello #1 " . date('c'));
sleep(1);
my_echo("<br/>Hello #2 " . date('c'));
sleep(2);
my_echo("<br/>Hello #3 " . date('c'));
sleep(3);
my_echo("<br/>Hello #4 " . date('c'));
sleep(4);
my_echo("<br/>Hello #5 " . date('c'));
sleep(1);
my_echo("<br/>Hello #6 " . date('c'));

Open in new window

0
 
LVL 1

Author Comment

by:movieprodw
ID: 40441009
Hello,

Okay, I will try this out, it is a bummer you can not just set a timeout, it times out at 7 minutes ~420 seconds no matter what I set the max at.

Matt
0
What does it mean to be "Always On"?

Is your cloud always on? With an Always On cloud you won't have to worry about downtime for maintenance or software application code updates, ensuring that your bottom line isn't affected.

 
LVL 1

Author Comment

by:movieprodw
ID: 40441013
You can see the settings here, I set the max huge for testing

http://makemodelsearch.com/trash2.php
0
 
LVL 36

Expert Comment

by:Kimputer
ID: 40442177
You still didn't mention the httpd.conf Timeout setting.
0
 
LVL 1

Author Comment

by:movieprodw
ID: 40442940
#
# Timeout: The number of seconds before receives and sends time out.
#
Timeout 60
0
 
LVL 36

Expert Comment

by:Kimputer
ID: 40445021
Can you try Timeout 600 (restart httpd) and see if it gets better?
0
 
LVL 1

Author Closing Comment

by:movieprodw
ID: 40480612
Worked, thanks
0
 
LVL 36

Expert Comment

by:Kimputer
ID: 40480617
What was the solution then?
0

Featured Post

Get MySQL database support online, now!

At Percona’s web store you can order your MySQL database support needs in minutes. No hassles, no fuss, just pick and click. Pay online with a credit card.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Google Drive is extremely cheap offsite storage, and it's even possible to get extra storage for free for two years.  You can use the free account 15GB, and if you have an Android device..when you install Google Drive for the first time it will give…
This article discusses how to implement server side field validation and display customized error messages to the client.
Learn how to get help with Linux/Unix bash shell commands. Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…
How to Install VMware Tools in Red Hat Enterprise Linux 6.4 (RHEL 6.4) Step-by-Step Tutorial
Suggested Courses

752 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question