Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people, just like you, are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
Solved

Server PHP script timeout 500 error

Posted on 2014-11-09
9
703 Views
Last Modified: 2014-12-04
Hello,

I have a php script that crawls a bunch of websites for new jobs, curl.

I am getting a timeout for a couple job titles as there are a lot of them on the other websites.

The only function of the server is to find these jobs so I am not concerned about the length of the script running.

I have tried to edit the fcgid module but it still times out around 9 minutes... any where else you suggest or anything you see here?

# This is the Apache server configuration file for providing FastCGI support
# via mod_fcgid
#
# Documentation is available at http://fastcgi.coremail.cn/doc.htm

LoadModule fcgid_module modules/mod_fcgid.so

<IfModule mod_fcgid.c>

<IfModule !mod_fastcgi.c>
    AddHandler fcgid-script fcg fcgi fpl
</IfModule>

  FcgidIPCDir /var/run/mod_fcgid/sock
  FcgidProcessTableFile /var/run/mod_fcgid/fcgid_shm

  FcgidIdleTimeout 40
  FcgidProcessLifeTime 1200
  FcgidMaxProcesses 40
  FcgidMaxProcessesPerClass 8
  FcgidMinProcessesPerClass 0
  FcgidConnectTimeout 1200
  FcgidIOTimeout 1200
  FcgidInitialEnv RAILS_ENV production
  FcgidIdleScanInterval 10

</IfModule>

Open in new window

0
Comment
Question by:movieprodw
  • 4
  • 4
9 Comments
 
LVL 35

Expert Comment

by:Kimputer
ID: 40432234
You should look at php.ini for
max_execution_time (in seconds)
and also apache's httpd.conf parameter TimeOut (in seconds)

Or you should just run the script from commandline instead of web interface
0
 
LVL 109

Accepted Solution

by:
Ray Paseur earned 500 total points
ID: 40432253
A solution that I've found to a similar problem goes something like this:

1. Before each iteration in the long-running process, perhaps before you use cURL to acquire the HTML documents from the other sites, use set_time_limit(XX) with a value in XX that makes sense for the duration of that step of the process.  It might be 5 or 15 seconds - whatever you need to get done with that step.  Don't use 0 since an errant script loop will never be aborted.

2. At the same time, produce some amount of browser output.  You may be able to just echo the name of the URL, or something like that.  Then use flush() to cause PHP to poke the web server.  You may find that you need to pad the output in order to make it long enough that it gets past PHP and server buffering and actually gets sent to the browser.

Taken together these two things will (1) keep the server from killing the PHP script prematurely and (2) keep the server from thinking the PHP script has gone to sleep and (3) give you a browser-visual indicator of the process.  You may want to read the man pages for flush() carefully; it's not 100% intuitive.

Here's a simulation script that seems to work well for me:
http://iconoun.com/demo/force_flush.php

<?php // force_flush.php
error_reporting(E_ALL);
date_default_timezone_set('America/New_York');


// SIMULATE A LONG-RUNNING JOB WITH INTERMITTENT BROWSER OUTPUT
// FORCE FLUSH THE BROWSER OUTPUT BEFORE THE END OF THE PAGE
// PAD THE OUTPUT STRING TO MAKE IT LONG SO THE SERVER SENDS IT


function my_echo($str, $len=512, $pad=' ')
{
    // PAD THE STRING WITH INVISIBLE WHITESPACE OR SIMILAR
    $str = str_pad($str, strlen($str) + $len, $pad);
    echo $str;
    echo PHP_EOL;
    flush();
}


// TEST THE URL ARGUMENT FOR "b=y" AND OPTIONALLY START THE OUTPUT BUFFER
$b = (!empty($_GET["b"])) ? $_GET["b"] : NULL;
if ($b == 'y') ob_start();


// RUN THE TEST MESSAGES, WAITING BETWEEN THE MESSAGES
my_echo("<br/>Hello #1 " . date('c'));
sleep(1);
my_echo("<br/>Hello #2 " . date('c'));
sleep(2);
my_echo("<br/>Hello #3 " . date('c'));
sleep(3);
my_echo("<br/>Hello #4 " . date('c'));
sleep(4);
my_echo("<br/>Hello #5 " . date('c'));
sleep(1);
my_echo("<br/>Hello #6 " . date('c'));

Open in new window

0
 
LVL 1

Author Comment

by:movieprodw
ID: 40441009
Hello,

Okay, I will try this out, it is a bummer you can not just set a timeout, it times out at 7 minutes ~420 seconds no matter what I set the max at.

Matt
0
PRTG Network Monitor: Intuitive Network Monitoring

Network Monitoring is essential to ensure that computer systems and network devices are running. Use PRTG to monitor LANs, servers, websites, applications and devices, bandwidth, virtual environments, remote systems, IoT, and many more. PRTG is easy to set up & use.

 
LVL 1

Author Comment

by:movieprodw
ID: 40441013
You can see the settings here, I set the max huge for testing

http://makemodelsearch.com/trash2.php
0
 
LVL 35

Expert Comment

by:Kimputer
ID: 40442177
You still didn't mention the httpd.conf Timeout setting.
0
 
LVL 1

Author Comment

by:movieprodw
ID: 40442940
#
# Timeout: The number of seconds before receives and sends time out.
#
Timeout 60
0
 
LVL 35

Expert Comment

by:Kimputer
ID: 40445021
Can you try Timeout 600 (restart httpd) and see if it gets better?
0
 
LVL 1

Author Closing Comment

by:movieprodw
ID: 40480612
Worked, thanks
0
 
LVL 35

Expert Comment

by:Kimputer
ID: 40480617
What was the solution then?
0

Featured Post

Free Tool: ZipGrep

ZipGrep is a utility that can list and search zip (.war, .ear, .jar, etc) archives for text patterns, without the need to extract the archive's contents.

One of a set of tools we're offering as a way to say thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

I. Introduction There's an interesting discussion going on now in an Experts Exchange Group — Attachments with no extension (http://www.experts-exchange.com/discussions/210281/Attachments-with-no-extension.html). This reminded me of questions tha…
Password hashing is better than message digests or encryption, and you should be using it instead of message digests or encryption.  Find out why and how in this article, which supplements the original article on PHP Client Registration, Login, Logo…
Learn how to navigate the file tree with the shell. Use pwd to print the current working directory: Use ls to list a directory's contents: Use cd to change to a new directory: Use wildcards instead of typing out long directory names: Use ../ to move…
Connecting to an Amazon Linux EC2 Instance from Windows Using PuTTY.

840 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question