Solved

Server PHP script timeout 500 error

Posted on 2014-11-09
9
613 Views
Last Modified: 2014-12-04
Hello,

I have a php script that crawls a bunch of websites for new jobs, curl.

I am getting a timeout for a couple job titles as there are a lot of them on the other websites.

The only function of the server is to find these jobs so I am not concerned about the length of the script running.

I have tried to edit the fcgid module but it still times out around 9 minutes... any where else you suggest or anything you see here?

# This is the Apache server configuration file for providing FastCGI support
# via mod_fcgid
#
# Documentation is available at http://fastcgi.coremail.cn/doc.htm

LoadModule fcgid_module modules/mod_fcgid.so

<IfModule mod_fcgid.c>

<IfModule !mod_fastcgi.c>
    AddHandler fcgid-script fcg fcgi fpl
</IfModule>

  FcgidIPCDir /var/run/mod_fcgid/sock
  FcgidProcessTableFile /var/run/mod_fcgid/fcgid_shm

  FcgidIdleTimeout 40
  FcgidProcessLifeTime 1200
  FcgidMaxProcesses 40
  FcgidMaxProcessesPerClass 8
  FcgidMinProcessesPerClass 0
  FcgidConnectTimeout 1200
  FcgidIOTimeout 1200
  FcgidInitialEnv RAILS_ENV production
  FcgidIdleScanInterval 10

</IfModule>

Open in new window

0
Comment
Question by:movieprodw
  • 4
  • 4
9 Comments
 
LVL 35

Expert Comment

by:Kimputer
Comment Utility
You should look at php.ini for
max_execution_time (in seconds)
and also apache's httpd.conf parameter TimeOut (in seconds)

Or you should just run the script from commandline instead of web interface
0
 
LVL 108

Accepted Solution

by:
Ray Paseur earned 500 total points
Comment Utility
A solution that I've found to a similar problem goes something like this:

1. Before each iteration in the long-running process, perhaps before you use cURL to acquire the HTML documents from the other sites, use set_time_limit(XX) with a value in XX that makes sense for the duration of that step of the process.  It might be 5 or 15 seconds - whatever you need to get done with that step.  Don't use 0 since an errant script loop will never be aborted.

2. At the same time, produce some amount of browser output.  You may be able to just echo the name of the URL, or something like that.  Then use flush() to cause PHP to poke the web server.  You may find that you need to pad the output in order to make it long enough that it gets past PHP and server buffering and actually gets sent to the browser.

Taken together these two things will (1) keep the server from killing the PHP script prematurely and (2) keep the server from thinking the PHP script has gone to sleep and (3) give you a browser-visual indicator of the process.  You may want to read the man pages for flush() carefully; it's not 100% intuitive.

Here's a simulation script that seems to work well for me:
http://iconoun.com/demo/force_flush.php

<?php // force_flush.php
error_reporting(E_ALL);
date_default_timezone_set('America/New_York');


// SIMULATE A LONG-RUNNING JOB WITH INTERMITTENT BROWSER OUTPUT
// FORCE FLUSH THE BROWSER OUTPUT BEFORE THE END OF THE PAGE
// PAD THE OUTPUT STRING TO MAKE IT LONG SO THE SERVER SENDS IT


function my_echo($str, $len=512, $pad=' ')
{
    // PAD THE STRING WITH INVISIBLE WHITESPACE OR SIMILAR
    $str = str_pad($str, strlen($str) + $len, $pad);
    echo $str;
    echo PHP_EOL;
    flush();
}


// TEST THE URL ARGUMENT FOR "b=y" AND OPTIONALLY START THE OUTPUT BUFFER
$b = (!empty($_GET["b"])) ? $_GET["b"] : NULL;
if ($b == 'y') ob_start();


// RUN THE TEST MESSAGES, WAITING BETWEEN THE MESSAGES
my_echo("<br/>Hello #1 " . date('c'));
sleep(1);
my_echo("<br/>Hello #2 " . date('c'));
sleep(2);
my_echo("<br/>Hello #3 " . date('c'));
sleep(3);
my_echo("<br/>Hello #4 " . date('c'));
sleep(4);
my_echo("<br/>Hello #5 " . date('c'));
sleep(1);
my_echo("<br/>Hello #6 " . date('c'));

Open in new window

0
 
LVL 1

Author Comment

by:movieprodw
Comment Utility
Hello,

Okay, I will try this out, it is a bummer you can not just set a timeout, it times out at 7 minutes ~420 seconds no matter what I set the max at.

Matt
0
 
LVL 1

Author Comment

by:movieprodw
Comment Utility
You can see the settings here, I set the max huge for testing

http://makemodelsearch.com/trash2.php
0
How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

 
LVL 35

Expert Comment

by:Kimputer
Comment Utility
You still didn't mention the httpd.conf Timeout setting.
0
 
LVL 1

Author Comment

by:movieprodw
Comment Utility
#
# Timeout: The number of seconds before receives and sends time out.
#
Timeout 60
0
 
LVL 35

Expert Comment

by:Kimputer
Comment Utility
Can you try Timeout 600 (restart httpd) and see if it gets better?
0
 
LVL 1

Author Closing Comment

by:movieprodw
Comment Utility
Worked, thanks
0
 
LVL 35

Expert Comment

by:Kimputer
Comment Utility
What was the solution then?
0

Featured Post

Free Trending Threat Insights Every Day

Enhance your security with threat intelligence from the web. Get trending threat insights on hackers, exploits, and suspicious IP addresses delivered to your inbox with our free Cyber Daily.

Join & Write a Comment

Part of the Global Positioning System A geocode (https://developers.google.com/maps/documentation/geocoding/) is the major subset of a GPS coordinate (http://en.wikipedia.org/wiki/Global_Positioning_System), the other parts being the altitude and t…
Join Greg Farro and Ethan Banks from Packet Pushers (http://packetpushers.net/podcast/podcasts/pq-show-93-smart-network-monitoring-paessler-sponsored/) and Greg Ross from Paessler (https://www.paessler.com/prtg) for a discussion about smart network …
Learn how to navigate the file tree with the shell. Use pwd to print the current working directory: Use ls to list a directory's contents: Use cd to change to a new directory: Use wildcards instead of typing out long directory names: Use ../ to move…
Connecting to an Amazon Linux EC2 Instance from Windows Using PuTTY.

772 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

12 Experts available now in Live!

Get 1:1 Help Now