Mid-Atlantic-Data
asked on
problem with php file completely finishing on godaddy shared deluxe linux site
Problem: Have a web application running on godaddy - shared linux deluxe
plan - using php4 and mysql.
(am totally uneducated in php, mysql and apache - a software
consultant did the project but is no longer available at all -
a second software consultant cannot find the problem)
(godaddy insists they can do nothing to help with a site's
software the site did themself; once they said it's your
htaccess file)
Once a day (early am) we run an import_both.php (via cron) job that rebuilds
parts of the mysql data files. import_both.php runs both
import_custdata.php and import_otherdata.php.
From end Dec 2009 thru Jun 17, 2010 the import_both.php ran
successfully most times - very rarely would the job not finish.
(The job displays at every 100th record loaded since initially
we had problems getting the job to complete.)
(we now display at every 500th record - just in case it
might help - didn't seem to)
Originally, it reloaded approximately 80,000 records in import_custdata.php
and close to 3,000 in import_otherdata.php.
Starting early on June 18, 2010 the files only load
completely at most 40% of the days.
In desperation, we removed records that are least used
so we are now loading somewhere around 50,000 records.
The input file size for the job is approximately 8MB.
The site is more used now than originally but it is still
not heavily used. We assume that most often no user would
be on the site very early in the am, but occasionally
someone could be on the site.
The email from godaddy would only show the echos of
each 100 records until it decided to stop.
BUT - we could also run the job via keying in its entire
address in Internet Explorer, entering the user name/password
and then watch it run. (https: ... (secure site))
often it would get:
Internal Server Error
The server encountered an internal error or misconfiguration and was unable to complete your request.
Please contact the server administrator, support@supportwebsite.com
More information about this error may be available in the server error log.
--------------------------
Apache/1.3.33 Server at unitedreaders.com Port 443
BUT - if we hit the 'refresh' key it would often finish.
Or we would try it multiple times and often one of those times would finish.
Have tried researching this and doing the suggestions, but still a problem.
Went into godaddy Hosting GUI, Settings then the File Extensions, and changed
the file association for PHP4 files from " PHP 4.xFastCGI" to just "PHP 4.x".
Now - we no longer get the 'Internal Server Error' message when we run
the job via Internet Explorer - it just seems to stall and not continue -
but usually hitting REFRESH seems to re-start it and then it often finishes.
When a user is on the site, there are coding errors
(some missing files) that will go into godaddy's log
(if we set it on godaddy's site to create a log), but
these errors seem mostly not to affect the user - they
just do not see a certain 'picture' or icon.
Here is the php.ini file that is in the directory that holds the php
file that sometimes finishes, sometimes doesn't:
(many of the directories have an .htaccess file and php.ini file -
that are different than these - e.g. the one the user visiting
the site gets to).
register_globals = off
allow_url_fopen = off
expose_php = Off
max_input_time = 1800
max_execution_time = 1800
mysql.connect_timeout = 240
memory_limit = 128M
post_max_size = 64000000
upload_max_filesize = 64000000
variables_order = "EGPCS"
extension_dir = ./
upload_tmp_dir = /tmp
precision = 12
SMTP = relay-hosting.secureserver
url_rewriter.tags = "a=href,area=href,frame=sr
[Zend]
zend_extension=/usr/local/
zend_extension=/usr/local/
Here is the .htaccess file that is in the directory that holds the php
file that sometimes finishes, sometimes doesn't:
authuserfile /home/dir1/dir2/dir3/dir4/
authtype basic
authgroupfile /dev/null
authname "Secure Area"
require user admin1 admin2 admin3
ANY help would be MOST appreciated. Unfortunately, the true testing
of any changes can only happen once a day - and about 40% of the time
the job does run successfully.
ASKER
good suggestion to create a test environment .... even tho our skills are minimal.
Re job timing out - in the php.ini file - commads max_input_time
and max_execute_time seem more than sufficient - might it be worth it
to increase them?? or are there other commands also that affect time?
re running the job 4 times - the import_custdata.php completely replaces all the custdata records - so if the 4th (or last) time fails, then we are left with a partially incomplete website - not acceptable. That's why - if the cron job doesn't complete,
we run it via Internet Explorer until it does complete. But this is not really satisfactory to do 7 days a week.
Since starting to run only php4 - not php4-FastCGI - when run the job via Internet Explorer, hitting the REFRESH key starts the echo of the 100th (or 500th) record load point all over again - at least on the screen display.
Re job timing out - in the php.ini file - commads max_input_time
and max_execute_time seem more than sufficient - might it be worth it
to increase them?? or are there other commands also that affect time?
re running the job 4 times - the import_custdata.php completely replaces all the custdata records - so if the 4th (or last) time fails, then we are left with a partially incomplete website - not acceptable. That's why - if the cron job doesn't complete,
we run it via Internet Explorer until it does complete. But this is not really satisfactory to do 7 days a week.
Since starting to run only php4 - not php4-FastCGI - when run the job via Internet Explorer, hitting the REFRESH key starts the echo of the 100th (or 500th) record load point all over again - at least on the screen display.
If the problem does turn out to be running out of time, then re-writing it to do smaller pieces every five minutes until it gets it done is a way of getting around that.
I am curious why your are replacing the entire database every day. And you might consider this an opportunity to upgrade to PHP 5 while you're at it. You're probably going to have to hire a consultant to make some changes any way.
I am curious why your are replacing the entire database every day. And you might consider this an opportunity to upgrade to PHP 5 while you're at it. You're probably going to have to hire a consultant to make some changes any way.
ASKER
ok - you haven't specifically said to increase those 2 max values, but I will for tonight.
Our main software runs on an older in-house software program.
easiest way to give customers some web access was just to replace all the fields they see every day - not deluxe,
but seemed to be quick and dirty and satisfactory - that is, until June 18.
Our main software runs on an older in-house software program.
easiest way to give customers some web access was just to replace all the fields they see every day - not deluxe,
but seemed to be quick and dirty and satisfactory - that is, until June 18.
There are a number of ways to address this problem. The first thing is that I doubt that all those values are changing every day. Do you know how much the data changes each day? If it's only a few percent then you could do updates instead of replacing the whole database. Another alternative is to use two sets of tables. At any given time, one is the working set and the other is the update set. When the update is done, you switch them.
I have several web sites on Godaddy. I always make my MySQL tables externally accessible. Then I can do updates remotely from my servers here in my office.
The first thing is to gather information. How much does the data change? How long does it take to actually do the updates?
I have several web sites on Godaddy. I always make my MySQL tables externally accessible. Then I can do updates remotely from my servers here in my office.
The first thing is to gather information. How much does the data change? How long does it take to actually do the updates?
ASKER
the data actually changes very little. The one table would be relatively simple to only send the updates. The other data changes in many places in our software. Would probably need to do a diff between the old fiile and the newest replacement and then only send the differences to godaddy.
Was very surprised that 80,000 records could be a problem.
since last suggestion, changed the php.ini file: the 2 max fields which were 1800 are now at 6000. the mysql one which was 240 is now 960.
Also changed the memory_limit from 128M to 32M - just in case the import stops if suddenly enough memory is not available ???
Tried is last night 30 min. after making the changes and it worked (of course, it works 40% of the time).
Worked again at the cron time a little while ago.
Have never timed the import when run via Internet explorer - but guess sometimes it's as fast as 10 seconds - other times may take a minute or so and then sometimes it just 'sits' and even after several minutes, get no more displays of every 100th record - so seems to have stopped without finishing.
Was very surprised that 80,000 records could be a problem.
since last suggestion, changed the php.ini file: the 2 max fields which were 1800 are now at 6000. the mysql one which was 240 is now 960.
Also changed the memory_limit from 128M to 32M - just in case the import stops if suddenly enough memory is not available ???
Tried is last night 30 min. after making the changes and it worked (of course, it works 40% of the time).
Worked again at the cron time a little while ago.
Have never timed the import when run via Internet explorer - but guess sometimes it's as fast as 10 seconds - other times may take a minute or so and then sometimes it just 'sits' and even after several minutes, get no more displays of every 100th record - so seems to have stopped without finishing.
Don't think of it as 80,000 records, think of 200 plus users on shared hosting also doing their work in the middle of the night. If you were the only one of the server, 80,000 records could take as long as 80 seconds.
Until you're pretty sure it's been fixed, you need to keep a log of what time the update is done, how long it took, and did it finish. In addition, you might want to do something to see when the update data is actually available. Figure out what the last record is and make up a query to get that one. Godaddy's MySQL servers are on different machines than the web servers.
Until you're pretty sure it's been fixed, you need to keep a log of what time the update is done, how long it took, and did it finish. In addition, you might want to do something to see when the update data is actually available. Figure out what the last record is and make up a query to get that one. Godaddy's MySQL servers are on different machines than the web servers.
ASKER
your initial thought seemed to be it ran out of time ..
now max_execution_time = 6000
max_input_time = 6000
mysql.connect_timeout = 960
. what do you think re increasing those maxes??
. have no idea what mysql.connect_timeout really does -> what do you think of the 960 value?
. have been logging whether job completed or not and what time the email from godaddy is time stamped - but would need to change the import...php file to
somehow echo the actual start and end time. Really don't know how to do that.
now max_execution_time = 6000
max_input_time = 6000
mysql.connect_timeout = 960
. what do you think re increasing those maxes??
. have no idea what mysql.connect_timeout really does -> what do you think of the 960 value?
. have been logging whether job completed or not and what time the email from godaddy is time stamped - but would need to change the import...php file to
somehow echo the actual start and end time. Really don't know how to do that.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
You obviously have a lot of successful experience in these matters.
We will consider doing updates only - but it will take a bit for us to switch
the software both internally and on the website.
re alternating tables - no idea how to do that ...
In the meantime, we have 2 successful imports completed - will see what the
next several days bring ...
If they are successful, we will accept the solution.
... may even try the 80,000 records we really want rather than the 50,000 we are getting by with.
Thank you VERY much for all your suggestions!
We will consider doing updates only - but it will take a bit for us to switch
the software both internally and on the website.
re alternating tables - no idea how to do that ...
In the meantime, we have 2 successful imports completed - will see what the
next several days bring ...
If they are successful, we will accept the solution.
... may even try the 80,000 records we really want rather than the 50,000 we are getting by with.
Thank you VERY much for all your suggestions!
Glad to help. Most of my experience is in figuring how things get screwed up and what to do next. Most of us test our databases on our own servers... which don't have 100's of other users.
If you're expecting growth on your website, you should still make a longer term plan to make sure the updates get completed. There are a variety of software methods for doing that.
If you're expecting growth on your website, you should still make a longer term plan to make sure the updates get completed. There are a variety of software methods for doing that.
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
job failed again - this time after 8000 records, but before 8500
Does your email tell you the start and stop times so you'll know how long it was active?
ASKER
it doesn't .
But it does have the email time stamped and the cron job is always set to run at the same time every day.
Could not find any consistent correlation between them.
e.g. the incomplete import of 8000+ records is email time stamped 1 min after the cron job's schedule time.
So it seems the job had to only be running under 2 minutes.
But it does have the email time stamped and the cron job is always set to run at the same time every day.
Could not find any consistent correlation between them.
e.g. the incomplete import of 8000+ records is email time stamped 1 min after the cron job's schedule time.
So it seems the job had to only be running under 2 minutes.
Save the file below as 'phpchk.php' and run it on your web site. Let me know what it says for 'max_execution_time'. On two of my Godaddy sites it is 30 seconds.
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<title>PHP Check</title>
</head>
<body>
<p>PHP Check</p>
<?php phpinfo() ?>
<?php phpversion() ?>
</body>
</html>
ASKER
Ran the file from my import dir
Max_execution_time 1800
This is the max_execution_time in the 'root' directory,
but it is now set to 9000 in the import directory
Max_execution_time 1800
This is the max_execution_time in the 'root' directory,
but it is now set to 9000 in the import directory
Since the last failed job apparently ran less than 120 seconds (two minutes), that's probably not going to make any difference. I think it's clear that you need a better method to make sure the updates get completed.
ASKER
re birwin comment of large import of 3.8 million records - we used to do
80,000 with no problem - and that's a lot less - but still not working consistently
now with 50,000 records - but appreciate the thought that we could be trying to do the impossible with godaddy.
re only doing the records that changed rather than replacing the whole file,
upon reflection - realized that once a month approximately 50% of the records
change on 1 day (The other days relatively few change). But that day would always be a problem. When we first began the website, we quickly switched to replacing the entire file because the update of individual records seemed to be way too time consuming. Could program around doing different jobs on different dates, but would much rather not.
The website is getting errors when a user accesses it because a few files called in the code are missing. We are in the process of correcting that.
'"robots.txt" file does not exist' is not uncommon. We are going to try and eliminate that error.
?? if the cron job doing the file update is in progress and a visitor to the site does something to cause an error that can go to the log file (like ones mentioned above), is it possible that could cause the import to halt ???
80,000 with no problem - and that's a lot less - but still not working consistently
now with 50,000 records - but appreciate the thought that we could be trying to do the impossible with godaddy.
re only doing the records that changed rather than replacing the whole file,
upon reflection - realized that once a month approximately 50% of the records
change on 1 day (The other days relatively few change). But that day would always be a problem. When we first began the website, we quickly switched to replacing the entire file because the update of individual records seemed to be way too time consuming. Could program around doing different jobs on different dates, but would much rather not.
The website is getting errors when a user accesses it because a few files called in the code are missing. We are in the process of correcting that.
'"robots.txt" file does not exist' is not uncommon. We are going to try and eliminate that error.
?? if the cron job doing the file update is in progress and a visitor to the site does something to cause an error that can go to the log file (like ones mentioned above), is it possible that could cause the import to halt ???
ASKER
Thank you both for your help - it just really seems that godaddy cannot consistently handle this amount of data (which really doesn't seem that large too me).
So either load the data differently or choose another provider.
So either load the data differently or choose another provider.
I'm afraid you're right. Thanks for the points.
The mostly likely thing is that since it is a shared hosting, other people's sites take up part of the CPU time and your job times out. If that turns out to be the case, then scheduling the cron job to run something like 4 times five minutes apart in the middle of the night might fix the problem. That might require a little re-write though. In doing a 'refresh', you are probably still in the 'session' for that file.