PHP File Uploading

I need some help as fast as possible, this is very urgent. I have a PHP script that takes a file upload, opens it and parses it into a MySQL DB.
The file to be uploaded is expected to be around 35 megabytes. I tested the script with a smaller file and it worked fine. Trying it with the larger file, however, I get an error that it cant display the page, like the target script isnt there at all.

My php.ini settings arent the cause (unless theres a relevant setting that I missed).

upload_max_filesize is large enough to accomodate the file and post_max_size is large enough to accomodate several of these files at once.

The script sets the maximum execution time and it has no limit.

Any help will be GREATLY appreciated.
jmfreemaAsked:
Who is Participating?
 
pcaylorConnect With a Mentor Commented:
You aren't running a version of Apache before 1.3.26 are you?  If so, upgrade.  There was a bug that caused premature timeouts that was fixed in 1.3.26.

My only other thought is to set the Apache timeout value to something insanely large (Say 6000)  

I am assuming that it's an Apache problem as you seem to have a pretty good handle on the PHP settings.

-Peter
0
 
gizmolaCommented:
If this is a post you also need to be concerned about post_max_size This value must be larger than upload_max_filesize.

Also if memory limit is enabled, generally speaking, memory_limit should be larger than post_max_size.

Last but not least there's max_execution_time which may be timing out the script before the upload can be completed.  Depending on whether or not you are running in safe mode, you may be able to use a function to set this for your upload script, or you may have to extend the allowed execution time to accomodate the large uploads.
0
 
jmfreemaAuthor Commented:
I already have post_max_size set higher than upload_max_size and both are set larger than I ever actually needed. I even stated as such in my original question. memory_limit is enabled, and is set larger than post_max_size. max_execution_time isnt a factor since, as I already stated, it is set via script and has no limit to execution-time.

safe mode is not an issue.

I dont mean to be rude, but everything you said in your answer, I had already addressed in my question and stated that those settings were not at fault. I already tried all of those things and mentioned as such in my initial question.
Im increasing the points by 50 in the hopes I'll get a better answer, one that actually helps.
0
Upgrade your Question Security!

Your question, your audience. Choose who sees your identity—and your question—with question security.

 
gizmolaCommented:
Yes a couple of things you addressed in your message, and I redundantly addressed those, being part of a larger picture.  

However, noplace did you discuss memory_limit, or your safe_mode setting.  These type of problems are often only addressed through incremental investigation.

Other possible issues, are you setting the MAX_FILE_SIZE of your input object, prior to the File Input object ala:

<INPUT TYPE="hidden" name="MAX_FILE_SIZE" value="1000000">

The problem could very possibly be on your mysql side.  are you examining returns from the mysql functions or using @ to hide errors?  

You haven't provided any information about what platform you're running on or the version of Mysql.  If you have 3.23 you're not going to be able to work due to a limitation on the mysql client-server protocol level, which limits single statements to 16m or less.




0
 
gizmolaCommented:
In the MAX_FILE_SIZE example, obviously the value needs to be larger than I put in the example.  
0
 
jmfreemaAuthor Commented:
Im not able to view any errors. The page simply fails to load. It just gives me  

"The page cannot be found"
The page you are looking for might have been removed, had its name changed, or is temporarily unavailable."

etc.

It worked with the smaller files, but the larger files return that error. The form does not have a MAX_FILE_SIZE field, (mostly because I havent been told what the largest possible file expected to be used for upload), but the smaller (approximately 13 MB) works fine without it.

Also, the same script works fine when used on a local file of the same size, so the problem is clearly somewhere in the upload process and not in MySQL.

I appologize for my previous attitude, I was rather pressed for a solution and the stress got the better of me.
0
 
pcaylorCommented:
If the script works locally, could you try running the script through a web browser on the server? (Accessing it via http://localhost or local IP address)

If it works, then the problem is a timeout issue.  (Since the file upload will complete very quickly when done locally.)  If it does not work, then it is probably in the upload process.

My guess is it's timeout related.

-Peter
0
 
spookjeCommented:
Hi, just trying to help looking for the source of problems:

Did you check the mysql configration? ( /etc/my.cnf )

Default is:

[mysqld]
port=3306
socket=/tmp/mysql.sock
set-variable = key_buffer_size=16M
set-variable = max_allowed_packet=1M

If you dump all 32MB at once, maybe you get a conflict
with those vars?

/etc/init.d/mysqld restart #after changing conf

Next, it could be an apache problem ( /etc/httpd/conf/http.conf )
Apache decides to abort a request if too much time is spent...

#
# Timeout: The number of seconds before receives and sends time out.
#
Timeout 300

/etc/init.d/httpd restart #after changing conf

Moreover, is your 32MB partly processed, or rather not at all?

Finally, it can be the problem of your client falling asleep while you are POSTing your file, but that must be weird because it is busy doing something.

Also one side note: when you POST a file, it will probably encoded (base64 or such) because http doesn't allow you to send binaries. That means your encoded file may be of double size than it's origin (about 64MB?).

If you check those, maybe you find other possible sources for the problem by yourself? Anyway keep us informed if something more comes up that might help to find it!

-Spookje
0
 
jmfreemaAuthor Commented:
OK. I commented out all of the MySQL stuff because I wanted to see if it was MySQL or just the file upload. Even after the MySQL code was no longer in there, it was still hanging (showing a "Server not found" page, etc). So, MySQL is not to blame.

I set the timeout in apache higher, but its still bugging out. Looking at the apache documentation, (unless Im mis-reading it) timeout is how long to wait between packets and not total execution time for serving up the page. On the other hand, it did wait a little longer before dying when I changed that so I might experiment with higher values. Right now I have it at 600.

As for the previous post about going through a web browser on the server, I'm not entirely sure how to go about that.

My max values for upload file size and for post size and for memory limit etc. are at about 120M, so thats more than double the file size in case it is doing some funky encoding. It still isnt working.

Im totally at a loss
0
 
pcaylorCommented:
jmfreema,

My comment about testing through a web browser on the server is based on trying to narrow the problem down to a timeout vs. a code issue.

To test this, I would suggest copying the file you wish to upload to a directory somewhere on the server and run your code to "upload" it to the server just like you would if you were uploading from another computer.

Since the file is already on the server, the upload will run very quickly as it is essentially just copying the file from one location to another on the same server.

If this "upload" works, then the problem is almost certainly a timeout issue, since the only thing that has changed is the source of the file being uploaded.  (A fast local file instead of a file over a slower network link.)

I hope that clarifies it some and helps narrow down the potential cause.

-Peter
0
 
cscorbetCommented:
Have you checked the:

upload_max_filesize = xxx

in your php.ini ?
0
 
cscorbetCommented:
sorry you have :(
0
 
jmfreemaAuthor Commented:
OK. I commented out all of the MySQL stuff because I wanted to see if it was MySQL or just the file upload. Even after the MySQL code was no longer in there, it was still hanging (showing a "Server not found" page, etc). So, MySQL is not to blame.

I set the timeout in apache higher, but its still bugging out. Looking at the apache documentation, (unless Im mis-reading it) timeout is how long to wait between packets and not total execution time for serving up the page. On the other hand, it did wait a little longer before dying when I changed that so I might experiment with higher values. Right now I have it at 600.

As for the previous post about going through a web browser on the server, I'm not entirely sure how to go about that.

My max values for upload file size and for post size and for memory limit etc. are at about 120M, so thats more than double the file size in case it is doing some funky encoding. It still isnt working.

Im totally at a loss
0
 
jmfreemaAuthor Commented:
Oops. Sorry for the repost, I refreshed the page :P

So... would I write another PHP script to just move the file from one location to another? I"m not clear, still, on uploading the file from the server to the server.
0
 
jmfreemaAuthor Commented:
On another point, I dont think its a code issue since it works fine for some files, but not ones on the order of 35M in size.
0
 
pcaylorCommented:
You shouldn't have to write another PHP script.  The way your system is setup you have a client (a workstation with a web browser and a file to be uploaded) and a server (running PHP, Apache, mySQL)

Instead of trying to upload the file from a client to the server, do a test where you do all of the client steps from the server instead of a different computer.  Open up a web browser on the server, open up your page with the upload form and try to upload the 35 MB file.

The file will be copied from one location on the server to another, but as far as the PHP, Apache and your code is concerned, it would be as if it had been a user over the Internet.  (The only difference is that the upload will run faster because it's all on one computer.)

If this test works, the problem is a timeout.  (I strongly suspect that this test will work.)

-Peter  
0
 
jmfreemaAuthor Commented:
Assume the local upload works, how do I make it work remotely (i.e. uploading from somewhere other than the server). Assuming it is a timeout issue, then.
0
 
spookjeCommented:
Could you check your apache log and mysql log?
You know that apache default logs a 'success' code 200,
404 if file is not found etc, nothing if the request
was appearently not received etc.. Check the code! :-)

I asked earler btw... is the SQL-code (partly) executed?
Also that must be logged. If not configure mysql-server
to log more verbosely.

Log files on RedHat are resided in
/var/log/httpd/access_log
/var/log/mysqld.log

Do you use a linux-shell, and do you have curl installed?
Then instead using your favourite browser you could
try to use curl like a command line browser. It supports
cookies, can post, authentication, everything you could
ever need (RTFM :). That is to see if your browser is
the bad guy of course. curl usually wins, if you need
help, tell me how exactly how you need to post it.

And further .. of course you double checked all settings
you could find in phpinfo(). Take another look?

Another thing, but I think this should not exist:
maybe a firewall prevents you from letting through a
certain amount of data?

Trying to keep you busy... :)

-spookje
0
 
gizmolaCommented:
A couple things: first if you are using mysql prior to version 4.x you have a 16mb limit on objects that can be put into the database.

Also you should use the <INPUT TYPE="hidden" name="MAX_FILE_SIZE" value="bytes"> and make it something arbitrarily high.

Also, I'd set the apache timeout setting to be something like 1200.

0
 
gizmolaCommented:
Don't forget to restart the apache process after making any config settings to the php.ini or apache config files.
0
 
jmfreemaAuthor Commented:
Its not MySQL, that goes line by line to correct syntax and I seriously hope there isnt a single line over 16 MB. Plus, Im not worried about the MySQL end of it right now.

I guess the version of PHP could stand to be upgraded, as could apache, but I dont have the authority to do that so Im talking that over with the person/people who do.

Thans for the effort, guys... Im going to leave this question open for a while in case anyone has any new insights and/or I get everything upgraded and it still doesnt work.
0
 
gizmolaCommented:
I think you misunderstand.  Mysql can not accept a blob larger than 16mb, so even if you get other things working, you won't be able to put in a 25mb file.

Also, did you add the MAX_FILE_SIZE field, and did you bump up the apache timeout value?  
0
 
jmfreemaAuthor Commented:
The file isnt one blob but a very large text file containing individual records. The MySQL side of things works fine when the file is local, so theres nothing wrong with MySQL.

I did bump up the apache timeout value, perhaps not enough. Also, this version of Apache is pre 1.3.26, so the previously mentioned bug might be the cause. Im waiting to hear if an update is on the horizon.

MAX_FILE_SIZE didnt seem to have any effect.
0
 
jmfreemaAuthor Commented:
This did it. Thanks.
0
All Courses

From novice to tech pro — start learning today.