Still celebrating National IT Professionals Day with 3 months of free Premium Membership. Use Code ITDAY17

x
?
Solved

PHP File Uploading

Posted on 2003-03-06
24
Medium Priority
?
881 Views
Last Modified: 2012-08-13
I need some help as fast as possible, this is very urgent. I have a PHP script that takes a file upload, opens it and parses it into a MySQL DB.
The file to be uploaded is expected to be around 35 megabytes. I tested the script with a smaller file and it worked fine. Trying it with the larger file, however, I get an error that it cant display the page, like the target script isnt there at all.

My php.ini settings arent the cause (unless theres a relevant setting that I missed).

upload_max_filesize is large enough to accomodate the file and post_max_size is large enough to accomodate several of these files at once.

The script sets the maximum execution time and it has no limit.

Any help will be GREATLY appreciated.
0
Comment
Question by:jmfreema
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 10
  • 6
  • 4
  • +2
24 Comments
 
LVL 4

Expert Comment

by:gizmola
ID: 8084157
If this is a post you also need to be concerned about post_max_size This value must be larger than upload_max_filesize.

Also if memory limit is enabled, generally speaking, memory_limit should be larger than post_max_size.

Last but not least there's max_execution_time which may be timing out the script before the upload can be completed.  Depending on whether or not you are running in safe mode, you may be able to use a function to set this for your upload script, or you may have to extend the allowed execution time to accomodate the large uploads.
0
 

Author Comment

by:jmfreema
ID: 8084214
I already have post_max_size set higher than upload_max_size and both are set larger than I ever actually needed. I even stated as such in my original question. memory_limit is enabled, and is set larger than post_max_size. max_execution_time isnt a factor since, as I already stated, it is set via script and has no limit to execution-time.

safe mode is not an issue.

I dont mean to be rude, but everything you said in your answer, I had already addressed in my question and stated that those settings were not at fault. I already tried all of those things and mentioned as such in my initial question.
Im increasing the points by 50 in the hopes I'll get a better answer, one that actually helps.
0
 
LVL 4

Expert Comment

by:gizmola
ID: 8084770
Yes a couple of things you addressed in your message, and I redundantly addressed those, being part of a larger picture.  

However, noplace did you discuss memory_limit, or your safe_mode setting.  These type of problems are often only addressed through incremental investigation.

Other possible issues, are you setting the MAX_FILE_SIZE of your input object, prior to the File Input object ala:

<INPUT TYPE="hidden" name="MAX_FILE_SIZE" value="1000000">

The problem could very possibly be on your mysql side.  are you examining returns from the mysql functions or using @ to hide errors?  

You haven't provided any information about what platform you're running on or the version of Mysql.  If you have 3.23 you're not going to be able to work due to a limitation on the mysql client-server protocol level, which limits single statements to 16m or less.




0
Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 
LVL 4

Expert Comment

by:gizmola
ID: 8084776
In the MAX_FILE_SIZE example, obviously the value needs to be larger than I put in the example.  
0
 

Author Comment

by:jmfreema
ID: 8086133
Im not able to view any errors. The page simply fails to load. It just gives me  

"The page cannot be found"
The page you are looking for might have been removed, had its name changed, or is temporarily unavailable."

etc.

It worked with the smaller files, but the larger files return that error. The form does not have a MAX_FILE_SIZE field, (mostly because I havent been told what the largest possible file expected to be used for upload), but the smaller (approximately 13 MB) works fine without it.

Also, the same script works fine when used on a local file of the same size, so the problem is clearly somewhere in the upload process and not in MySQL.

I appologize for my previous attitude, I was rather pressed for a solution and the stress got the better of me.
0
 
LVL 1

Expert Comment

by:pcaylor
ID: 8088312
If the script works locally, could you try running the script through a web browser on the server? (Accessing it via http://localhost or local IP address)

If it works, then the problem is a timeout issue.  (Since the file upload will complete very quickly when done locally.)  If it does not work, then it is probably in the upload process.

My guess is it's timeout related.

-Peter
0
 
LVL 1

Expert Comment

by:spookje
ID: 8089403
Hi, just trying to help looking for the source of problems:

Did you check the mysql configration? ( /etc/my.cnf )

Default is:

[mysqld]
port=3306
socket=/tmp/mysql.sock
set-variable = key_buffer_size=16M
set-variable = max_allowed_packet=1M

If you dump all 32MB at once, maybe you get a conflict
with those vars?

/etc/init.d/mysqld restart #after changing conf

Next, it could be an apache problem ( /etc/httpd/conf/http.conf )
Apache decides to abort a request if too much time is spent...

#
# Timeout: The number of seconds before receives and sends time out.
#
Timeout 300

/etc/init.d/httpd restart #after changing conf

Moreover, is your 32MB partly processed, or rather not at all?

Finally, it can be the problem of your client falling asleep while you are POSTing your file, but that must be weird because it is busy doing something.

Also one side note: when you POST a file, it will probably encoded (base64 or such) because http doesn't allow you to send binaries. That means your encoded file may be of double size than it's origin (about 64MB?).

If you check those, maybe you find other possible sources for the problem by yourself? Anyway keep us informed if something more comes up that might help to find it!

-Spookje
0
 

Author Comment

by:jmfreema
ID: 8090185
OK. I commented out all of the MySQL stuff because I wanted to see if it was MySQL or just the file upload. Even after the MySQL code was no longer in there, it was still hanging (showing a "Server not found" page, etc). So, MySQL is not to blame.

I set the timeout in apache higher, but its still bugging out. Looking at the apache documentation, (unless Im mis-reading it) timeout is how long to wait between packets and not total execution time for serving up the page. On the other hand, it did wait a little longer before dying when I changed that so I might experiment with higher values. Right now I have it at 600.

As for the previous post about going through a web browser on the server, I'm not entirely sure how to go about that.

My max values for upload file size and for post size and for memory limit etc. are at about 120M, so thats more than double the file size in case it is doing some funky encoding. It still isnt working.

Im totally at a loss
0
 
LVL 1

Expert Comment

by:pcaylor
ID: 8090247
jmfreema,

My comment about testing through a web browser on the server is based on trying to narrow the problem down to a timeout vs. a code issue.

To test this, I would suggest copying the file you wish to upload to a directory somewhere on the server and run your code to "upload" it to the server just like you would if you were uploading from another computer.

Since the file is already on the server, the upload will run very quickly as it is essentially just copying the file from one location to another on the same server.

If this "upload" works, then the problem is almost certainly a timeout issue, since the only thing that has changed is the source of the file being uploaded.  (A fast local file instead of a file over a slower network link.)

I hope that clarifies it some and helps narrow down the potential cause.

-Peter
0
 
LVL 1

Expert Comment

by:cscorbet
ID: 8090292
Have you checked the:

upload_max_filesize = xxx

in your php.ini ?
0
 
LVL 1

Expert Comment

by:cscorbet
ID: 8090307
sorry you have :(
0
 

Author Comment

by:jmfreema
ID: 8090323
OK. I commented out all of the MySQL stuff because I wanted to see if it was MySQL or just the file upload. Even after the MySQL code was no longer in there, it was still hanging (showing a "Server not found" page, etc). So, MySQL is not to blame.

I set the timeout in apache higher, but its still bugging out. Looking at the apache documentation, (unless Im mis-reading it) timeout is how long to wait between packets and not total execution time for serving up the page. On the other hand, it did wait a little longer before dying when I changed that so I might experiment with higher values. Right now I have it at 600.

As for the previous post about going through a web browser on the server, I'm not entirely sure how to go about that.

My max values for upload file size and for post size and for memory limit etc. are at about 120M, so thats more than double the file size in case it is doing some funky encoding. It still isnt working.

Im totally at a loss
0
 

Author Comment

by:jmfreema
ID: 8090333
Oops. Sorry for the repost, I refreshed the page :P

So... would I write another PHP script to just move the file from one location to another? I"m not clear, still, on uploading the file from the server to the server.
0
 

Author Comment

by:jmfreema
ID: 8090348
On another point, I dont think its a code issue since it works fine for some files, but not ones on the order of 35M in size.
0
 
LVL 1

Expert Comment

by:pcaylor
ID: 8090542
You shouldn't have to write another PHP script.  The way your system is setup you have a client (a workstation with a web browser and a file to be uploaded) and a server (running PHP, Apache, mySQL)

Instead of trying to upload the file from a client to the server, do a test where you do all of the client steps from the server instead of a different computer.  Open up a web browser on the server, open up your page with the upload form and try to upload the 35 MB file.

The file will be copied from one location on the server to another, but as far as the PHP, Apache and your code is concerned, it would be as if it had been a user over the Internet.  (The only difference is that the upload will run faster because it's all on one computer.)

If this test works, the problem is a timeout.  (I strongly suspect that this test will work.)

-Peter  
0
 

Author Comment

by:jmfreema
ID: 8090727
Assume the local upload works, how do I make it work remotely (i.e. uploading from somewhere other than the server). Assuming it is a timeout issue, then.
0
 
LVL 1

Accepted Solution

by:
pcaylor earned 1100 total points
ID: 8091065
You aren't running a version of Apache before 1.3.26 are you?  If so, upgrade.  There was a bug that caused premature timeouts that was fixed in 1.3.26.

My only other thought is to set the Apache timeout value to something insanely large (Say 6000)  

I am assuming that it's an Apache problem as you seem to have a pretty good handle on the PHP settings.

-Peter
0
 
LVL 1

Expert Comment

by:spookje
ID: 8091130
Could you check your apache log and mysql log?
You know that apache default logs a 'success' code 200,
404 if file is not found etc, nothing if the request
was appearently not received etc.. Check the code! :-)

I asked earler btw... is the SQL-code (partly) executed?
Also that must be logged. If not configure mysql-server
to log more verbosely.

Log files on RedHat are resided in
/var/log/httpd/access_log
/var/log/mysqld.log

Do you use a linux-shell, and do you have curl installed?
Then instead using your favourite browser you could
try to use curl like a command line browser. It supports
cookies, can post, authentication, everything you could
ever need (RTFM :). That is to see if your browser is
the bad guy of course. curl usually wins, if you need
help, tell me how exactly how you need to post it.

And further .. of course you double checked all settings
you could find in phpinfo(). Take another look?

Another thing, but I think this should not exist:
maybe a firewall prevents you from letting through a
certain amount of data?

Trying to keep you busy... :)

-spookje
0
 
LVL 4

Expert Comment

by:gizmola
ID: 8092124
A couple things: first if you are using mysql prior to version 4.x you have a 16mb limit on objects that can be put into the database.

Also you should use the <INPUT TYPE="hidden" name="MAX_FILE_SIZE" value="bytes"> and make it something arbitrarily high.

Also, I'd set the apache timeout setting to be something like 1200.

0
 
LVL 4

Expert Comment

by:gizmola
ID: 8092125
Don't forget to restart the apache process after making any config settings to the php.ini or apache config files.
0
 

Author Comment

by:jmfreema
ID: 8093908
Its not MySQL, that goes line by line to correct syntax and I seriously hope there isnt a single line over 16 MB. Plus, Im not worried about the MySQL end of it right now.

I guess the version of PHP could stand to be upgraded, as could apache, but I dont have the authority to do that so Im talking that over with the person/people who do.

Thans for the effort, guys... Im going to leave this question open for a while in case anyone has any new insights and/or I get everything upgraded and it still doesnt work.
0
 
LVL 4

Expert Comment

by:gizmola
ID: 8094587
I think you misunderstand.  Mysql can not accept a blob larger than 16mb, so even if you get other things working, you won't be able to put in a 25mb file.

Also, did you add the MAX_FILE_SIZE field, and did you bump up the apache timeout value?  
0
 

Author Comment

by:jmfreema
ID: 8096278
The file isnt one blob but a very large text file containing individual records. The MySQL side of things works fine when the file is local, so theres nothing wrong with MySQL.

I did bump up the apache timeout value, perhaps not enough. Also, this version of Apache is pre 1.3.26, so the previously mentioned bug might be the cause. Im waiting to hear if an update is on the horizon.

MAX_FILE_SIZE didnt seem to have any effect.
0
 

Author Comment

by:jmfreema
ID: 8130716
This did it. Thanks.
0

Featured Post

Free Tool: Port Scanner

Check which ports are open to the outside world. Helps make sure that your firewall rules are working as intended.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

These days socially coordinated efforts have turned into a critical requirement for enterprises.
There are times when I have encountered the need to decompress a response from a PHP request. This is how it's done, but you must have control of the request and you can set the Accept-Encoding header.
Learn how to match and substitute tagged data using PHP regular expressions. Demonstrated on Windows 7, but also applies to other operating systems. Demonstrated technique applies to PHP (all versions) and Firefox, but very similar techniques will w…
The viewer will learn how to look for a specific file type in a local or remote server directory using PHP.

721 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question