Server PHP script timeout 500 error

Hello,

I have a php script that crawls a bunch of websites for new jobs, curl.

I am getting a timeout for a couple job titles as there are a lot of them on the other websites.

The only function of the server is to find these jobs so I am not concerned about the length of the script running.

I have tried to edit the fcgid module but it still times out around 9 minutes... any where else you suggest or anything you see here?

# This is the Apache server configuration file for providing FastCGI support
# via mod_fcgid
#
# Documentation is available at http://fastcgi.coremail.cn/doc.htm

LoadModule fcgid_module modules/mod_fcgid.so

<IfModule mod_fcgid.c>

<IfModule !mod_fastcgi.c>
    AddHandler fcgid-script fcg fcgi fpl
</IfModule>

  FcgidIPCDir /var/run/mod_fcgid/sock
  FcgidProcessTableFile /var/run/mod_fcgid/fcgid_shm

  FcgidIdleTimeout 40
  FcgidProcessLifeTime 1200
  FcgidMaxProcesses 40
  FcgidMaxProcessesPerClass 8
  FcgidMinProcessesPerClass 0
  FcgidConnectTimeout 1200
  FcgidIOTimeout 1200
  FcgidInitialEnv RAILS_ENV production
  FcgidIdleScanInterval 10

</IfModule>

Open in new window

LVL 1
movieprodwAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

KimputerCommented:
You should look at php.ini for
max_execution_time (in seconds)
and also apache's httpd.conf parameter TimeOut (in seconds)

Or you should just run the script from commandline instead of web interface
0
Ray PaseurCommented:
A solution that I've found to a similar problem goes something like this:

1. Before each iteration in the long-running process, perhaps before you use cURL to acquire the HTML documents from the other sites, use set_time_limit(XX) with a value in XX that makes sense for the duration of that step of the process.  It might be 5 or 15 seconds - whatever you need to get done with that step.  Don't use 0 since an errant script loop will never be aborted.

2. At the same time, produce some amount of browser output.  You may be able to just echo the name of the URL, or something like that.  Then use flush() to cause PHP to poke the web server.  You may find that you need to pad the output in order to make it long enough that it gets past PHP and server buffering and actually gets sent to the browser.

Taken together these two things will (1) keep the server from killing the PHP script prematurely and (2) keep the server from thinking the PHP script has gone to sleep and (3) give you a browser-visual indicator of the process.  You may want to read the man pages for flush() carefully; it's not 100% intuitive.

Here's a simulation script that seems to work well for me:
http://iconoun.com/demo/force_flush.php

<?php // force_flush.php
error_reporting(E_ALL);
date_default_timezone_set('America/New_York');


// SIMULATE A LONG-RUNNING JOB WITH INTERMITTENT BROWSER OUTPUT
// FORCE FLUSH THE BROWSER OUTPUT BEFORE THE END OF THE PAGE
// PAD THE OUTPUT STRING TO MAKE IT LONG SO THE SERVER SENDS IT


function my_echo($str, $len=512, $pad=' ')
{
    // PAD THE STRING WITH INVISIBLE WHITESPACE OR SIMILAR
    $str = str_pad($str, strlen($str) + $len, $pad);
    echo $str;
    echo PHP_EOL;
    flush();
}


// TEST THE URL ARGUMENT FOR "b=y" AND OPTIONALLY START THE OUTPUT BUFFER
$b = (!empty($_GET["b"])) ? $_GET["b"] : NULL;
if ($b == 'y') ob_start();


// RUN THE TEST MESSAGES, WAITING BETWEEN THE MESSAGES
my_echo("<br/>Hello #1 " . date('c'));
sleep(1);
my_echo("<br/>Hello #2 " . date('c'));
sleep(2);
my_echo("<br/>Hello #3 " . date('c'));
sleep(3);
my_echo("<br/>Hello #4 " . date('c'));
sleep(4);
my_echo("<br/>Hello #5 " . date('c'));
sleep(1);
my_echo("<br/>Hello #6 " . date('c'));

Open in new window

0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
movieprodwAuthor Commented:
Hello,

Okay, I will try this out, it is a bummer you can not just set a timeout, it times out at 7 minutes ~420 seconds no matter what I set the max at.

Matt
0
Ultimate Tool Kit for Technology Solution Provider

Broken down into practical pointers and step-by-step instructions, the IT Service Excellence Tool Kit delivers expert advice for technology solution providers. Get your free copy now.

movieprodwAuthor Commented:
You can see the settings here, I set the max huge for testing

http://makemodelsearch.com/trash2.php
0
KimputerCommented:
You still didn't mention the httpd.conf Timeout setting.
0
movieprodwAuthor Commented:
#
# Timeout: The number of seconds before receives and sends time out.
#
Timeout 60
0
KimputerCommented:
Can you try Timeout 600 (restart httpd) and see if it gets better?
0
movieprodwAuthor Commented:
Worked, thanks
0
KimputerCommented:
What was the solution then?
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Apache Web Server

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.