phpftp, too much load? same with CURL?

Hi,

I created a question/topic about this in the php forums but was told to go here since it might be a problem with ftp.
http://www.experts-exchange.com/Web/Web_Languages/PHP/Q_21478595.html

My upload script worked well in the begining but started to make problems when the load/uploads increased - i'm currently getting average 1800 uploads per day (increasing fast). It dosent post any errors, just fail (blank page). Have searched the logs but havent been able to find anything related to this issue.

Started with phpftp and switched to CURL since this should be able to handle a higher load (was told), but keep getting 0 byte uploads upon bigger files (above 1 MB) sometimes it goes okay (correct filesize) but most times it's just 0 byte. Changed it so it upload to the main server instead of the fileserver, waiting on the results of this test.

It works this way:
A user upload a file (up to 60MB) using a standard upload forum, after upload (which always goes okay) the file is FTP'ed to the correct fileserver.
kgp43Asked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Gabriel OrozcoSolution ArchitectCommented:
you need to separate all steps, so you can recognize in which point is the problem.

are you telling us the upload goes well and you get the file in /tmp, and then when you start the ftp to the internal host, your file get lost?
kgp43Author Commented:
Correct.

The file from the upload form is beeing uploaded to the tmp folder correct (can echo the filesize). The error accure when I FTP it - this both count internal (on the same server) and to our external fileservers.

When I use phpftp I just get a blank page without any error messages.
When I use CURL it upload the file but bigger files has a filesize of 0 byte.

Please note, phpftp upload from time to time without problems (also files about 50MB) but receiving the blank page 60-70% when the file exceed 1-2MB.
CURL also upload files about 10MB which is okay, but the successrate is more like 5-10%.
Gabriel OrozcoSolution ArchitectCommented:
maybe it is a question of timeouts and filesizes.

check you have a .htaccess file on the directory where the upload page is, with something like

<ifModule mod_php4.c>
php_value max_execution_time 1200
php_value memory_limit 200M
php_value post_max_size 200M
php_value upload_max_filesize 200M
</ifModule>

this is to allow a maximun of 200Mb
Become a Microsoft Certified Solutions Expert

This course teaches how to install and configure Windows Server 2012 R2.  It is the first step on your path to becoming a Microsoft Certified Solutions Expert (MCSE).

kgp43Author Commented:
think I already have all those set very high - havent heard of memory_limit before though. Can it be a memory issue?
Will add the .htaccess file into the dir where the upload script is located
kgp43Author Commented:
Please note that it can upload files up to 60MB (which is max) from time to time without problems.
 - Getting avg 10 uploads per day of files above 50MB.
Gabriel OrozcoSolution ArchitectCommented:
mmhh...

this leads me in a different direction then

what is in the error_log of apache when you get a blank page?

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
kgp43Author Commented:
Added the info into the htaccess file but still receive uploads with 0 byte filesize (CURL)
kgp43Author Commented:
mmhh...
this leads me in a different direction then
what is in the error_log of apache when you get a blank page?

Where do I find that error_log?
(know how to use putty ect, just need the location of the log)
Gabriel OrozcoSolution ArchitectCommented:
you have apache running.
I assume you didn't changed anything

so the apache log should be located at /var/log/apache

inside you should find two logs: access_log and error_log (maybe name slightly different) you must check these in order to see why a blank page.
kgp43Author Commented:
Didnt changed anything, maby my servers admins did (dont know).

Cant find the apache folder in the log folder.
The content of the "log" folder:

drwxr-xr-x   13 root     root         4096 Jul  4 19:24 ./
drwxr-xr-x   27 root     root         4096 Jun 11 09:16 ../
lrwxrwxrwx    1 root     root           22 May 19 21:12 audit -> /var/log/audit.d/bin.0
drwx------    2 root     root         4096 May 19 21:12 audit.d/
drwxr-xr-x    3 root     root         4096 Apr 29 14:02 bandwidth/
-rw-------    1 root     root            0 Jul  3 04:02 boot.log
-rw-------    1 root     root         1244 Jul  3 04:02 boot.log.1
-rw-------    1 root     root          777 Jun 26 04:02 boot.log.2
-rw-------    1 root     root          342 Jun 19 04:02 boot.log.3
-rw-------    1 root     root         1490 Jun 12 04:02 boot.log.4
drwxr-xr-x    2 bin      bin          4096 Sep 16  2003 canna/
-rw-r--r--    1 root     root            0 May 19 06:52 checkphpv_error.log
-rw-r--r--    1 root     root          882 May 19 06:52 checkphpv_output.log
-rw-r--r--    1 root     root      2130577 Jul  4 19:24 chkservd.log
-rw-r--r--    1 root     root        67879 Apr 29 14:24 cpanel-install-thread0.log
-rw-r--r--    1 root     root       679552 Apr 29 14:23 cpanel-install-thread1.log
-rw-r--r--    1 root     root         1334 May 19 06:49 cpupdate.env
-rw-r--r--    1 root     root        20062 May 19 06:50 cpupdate.log
-rw-------    1 root     root       313606 Jul  4 19:25 cron
-rw-------    1 root     root      1347940 Jul  3 04:02 cron.1
-rw-------    1 root     root      1346427 Jun 26 04:02 cron.2
-rw-------    1 root     root      1346327 Jun 19 04:02 cron.3
-rw-------    1 root     root      1346716 Jun 12 04:02 cron.4
drwxr-xr-x    2 lp       sys          4096 Jul  3 04:02 cups/
drwx------    3 root     root         4096 Jul  4 19:25 dcpumon/
-rw-r--r--    1 root     root        14869 Apr 29 14:34 dmesg
-rw-r-----    1 mailnull mail       669787 Jul  4 19:24 exim_mainlog
-rw-r-----    1 mailnull mail       377907 Jul  3 04:02 exim_mainlog.1.gz
-rw-r-----    1 mailnull mail       345071 Jun 26 04:02 exim_mainlog.2.gz
-rw-r-----    1 mailnull mail       258007 Jun 19 04:02 exim_mainlog.3.gz
-rw-r-----    1 mailnull mail       288485 Jun 12 04:02 exim_mainlog.4.gz
-rw-r-----    1 mailnull mail            0 Jul  3 04:02 exim_paniclog
-rw-r-----    1 mailnull mail           36 Jun 26 04:02 exim_paniclog.1.gz
-rw-r-----    1 mailnull mail           36 Jun 19 04:02 exim_paniclog.2.gz
-rw-r-----    1 mailnull mail           36 Jun 12 04:02 exim_paniclog.3.gz
-rw-r-----    1 mailnull mail           36 Jun  5 04:02 exim_paniclog.4.gz
-rw-r-----    1 mailnull mail         4232 Jul  3 15:57 exim_rejectlog
-rw-r-----    1 mailnull mail         2597 Jul  2 22:44 exim_rejectlog.1.gz
-rw-r-----    1 mailnull mail         1663 Jun 25 09:40 exim_rejectlog.2.gz
-rw-r-----    1 mailnull mail         2555 Jun 19 02:36 exim_rejectlog.3.gz
-rw-r-----    1 mailnull mail         2419 Jun 11 17:46 exim_rejectlog.4.gz
drwx------    2 root     root         4096 May  1 04:02 httpd/
-rw-------    1 root     root        86041 Apr 29 14:34 ksyms.0
-rw-------    1 root     root        85572 Apr 29 12:58 ksyms.1
-r--------    1 root     root     19136220 Jul  4 19:20 lastlog
drwxr-xr-x    2 root     root         4096 Apr 29 14:02 mail/
-rw-------    1 root     root      1311403 Jul  4 19:24 maillog
-rw-------    1 root     root      4297380 Jul  3 04:00 maillog.1
-rw-------    1 root     root      2856517 Jun 26 03:59 maillog.2
-rw-------    1 root     root      3722621 Jun 19 04:02 maillog.3
-rw-------    1 root     root      4572802 Jun 12 04:00 maillog.4
drwxrwsr-x    2 root     mailman      4096 Jul  3 04:02 mailman/
-rw-------    1 root     root       633359 Jul  4 19:24 messages
-rw-------    1 root     root      1441029 Jul  3 04:02 messages.1
-rw-------    1 root     root      2492745 Jun 26 04:02 messages.2
-rw-------    1 root     root      2240957 Jun 19 04:02 messages.3
-rw-------    1 root     root      2297121 Jun 12 04:02 messages.4
-rw-rw----    1 mysql    root       241135 Jun 29 13:42 mysqld.log
-rw-r--r--    1 root     root        18124 Jul  4 04:02 rpmpkgs
-rw-r--r--    1 root     root        18124 Jul  2 04:02 rpmpkgs.1
-rw-r--r--    1 root     root        18124 Jun 25 04:02 rpmpkgs.2
-rw-r--r--    1 root     root        18124 Jun 18 04:02 rpmpkgs.3
-rw-r--r--    1 root     root        18117 Jun 11 04:02 rpmpkgs.4
drwx------    2 root     root         4096 Mar  3 18:35 samba/
-rw-------    1 root     root       421875 Jul  4 19:24 secure
-rw-------    1 root     root       706120 Jul  3 04:00 secure.1
-rw-------    1 root     root      1578737 Jun 26 03:56 secure.2
-rw-------    1 root     root      1266953 Jun 19 04:02 secure.3
-rw-------    1 root     root      1183893 Jun 12 04:00 secure.4
-rw-------    1 root     root            0 Jul  3 04:02 spooler
-rw-------    1 root     root            0 Jun 26 04:02 spooler.1
-rw-------    1 root     root            0 Jun 19 04:02 spooler.2
-rw-------    1 root     root            0 Jun 12 04:02 spooler.3
-rw-------    1 root     root            0 Jun  5 04:02 spooler.4
drwxr-x---    2 squid    squid        4096 Feb 16 14:52 squid/
-rw-r--r--    1 root     root        21949 Apr 29 14:32 stunnel-4.04-build.log
-rw-r--r--    1 root     root            0 Jul  3 04:02 up2date
-rw-r--r--    1 root     root         1061 Jun 30 03:41 up2date.1
-rw-r--r--    1 root     root         2635 Jun 23 01:30 up2date.2
-rw-r--r--    1 root     root         2342 Jun 16 02:21 up2date.3
-rw-r--r--    1 root     root         3948 Jun 11 10:49 up2date.4
drwxr-xr-x    2 root     root         4096 Aug 11  2003 vbox/
-rw-rw-r--    1 root     utmp       113280 Jul  4 19:20 wtmp
-rw-rw-r--    1 root     utmp       711936 Jul  1 03:45 wtmp.1
lrwxrwxrwx    1 root     root           41 May 17 08:59 xferlog -> ../../usr/local/apache/domlogs/ftpxferlog
-rw-r--r--    1 root     root            2 Jul  4 02:51 xferlog.offset
-rw-r--r--    1 root     root            2 Jul  4 19:24 xferlog.offsetftpsep
kgp43Author Commented:
Is this the one:
/usr/local/apache/logs/error_log ?
kgp43Author Commented:
hmmm, /etc/httpd/logs also have a error_log :P
(me is confused now :P)
kgp43Author Commented:
those logs are huge, anything specific to search for?
Gabriel OrozcoSolution ArchitectCommented:
a) they must not be huge. if they are right now, is because you left them grow unoticed.
b) you can stop apache, rename them, and start apache so you get new fresh logs. then at the moment you get a blank page, you should go directly there and see last log lines to see the error apache is sending when it cannot create the pages.

c) is your hard disk full or your swap? you can check the disk with the utility "df" and the swap with "free"
kgp43Author Commented:
didnt had a favicon so the logs was full of 404 errors.
cleaned the logs and waiting for new upload reports.

found this to monitor error_log realtime:
tail -f /usr/local/apache/logs/error_log
decoleurCommented:
could it be a permissions issue with the files being dropped into your temp folder having a different UID/GID than the process that is to move the files to their final resting place?
kgp43Author Commented:
the files are created but have a filesize of 0 bytes - dont know if that is normal in the error you describe above.
kgp43Author Commented:
Received two new uploads since the error_log was reset, both with 0 byte filesize.
This is what the error_log contain (except alot of 'file does not exist' errors):

[Tue Jul  5 02:36:07 2005] [error] PHP Warning:  MAX_FILE_SIZE of 25000 bytes exceeded - file [avatar=C:\\Documents and Settings\\milenk0\\My Documents\\My Pictures\\ICP\\milenko avatar.jpg] not savedDocuments\\My Pictures\\ICP\\milenko avatar.jpg] not saved in Unknown on line 0
[Tue Jul  5 01:14:59 2005] [error] PHP Warning:  MAX_FILE_SIZE of 62914560 bytes exceeded - file [upld_file=C:\\Documents and Settings\\Beulah Wils
[Tue Jul  5 02:28:24 2005] [error] PHP Warning:  session_destroy(): Session object destruction failed in /home/savefile/public_html/fileh
[Tue Jul  5 02:22:51 2005] [error] PHP Warning:  ob_start(): output handler 'ob_gzhandler' cannot be used after 'URL-Rewriter' in /home/provostp/pu
[Tue Jul  5 02:19:23 2005] [error] (2)No such file or directory: Incorrect permissions on webroot "/home/savefile/public_html/_vti_pvt" and webroot'$
[Tue Jul  5 01:13:20 2005] [error] [client 203.160.1.70] request failed: erroneous characters after protocol string: If-Modified-Since: Wed, 22 Jun
[Mon Jul  4 23:55:36 2005] [error] [client 203.160.1.70] request failed: erroneous characters after protocol string: If-Modifi
[Mon Jul  4 23:35:23 2005] [error] [client 70.68.79.4] Invalid method in request \\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\
[Mon Jul  4 23:34:05 2005] [error] PHP Warning:  POST Content-Length of 1130177048 bytes exceeds the limit of 209715200 bytes in Unknown on line 0
[Mon Jul  4 23:02:54 2005] [error] [client 140.117.170.156] Invalid method in request \\xe3Z
[Mon Jul  4 23:02:55 2005] [error] [client 140.117.170.156] Invalid URI in request GNUTELLA CONNECT/0.6
[Mon Jul  4 22:39:56 2005] [error] [client 203.160.1.70] request failed: erroneous characters after protocol string: If-Modified-Since: Wed, 22
decoleurCommented:
check you php.ini file, what is  MAX_FILE_SIZE set to?
kgp43Author Commented:
upload_max_filesize er 100MB
cant find MAX_FILE_SIZE in php.ini
kgp43Author Commented:
sometimes they can upload files about 50-60MB without problems, so I dont think that is the problem - not sure what this problem is about though.
decoleurCommented:
I wonder where MAX_FILE_SIZE is defined then... take a look at this thread dealing with php uploads: http://forums.devshed.com/t26027/s.html
granted it is for windows but the method sounds similar.
they also mention two php.ini variables:
 -> POST_MAX_SIZE and
-> UPLOAD_MAX_FILESIZE

are those not getting in your way?

I wonder if there isn't a quota of some sort letting the user who wants to upload a single file in a session work but the one who wants to upload multiple files within a single session will fail.

HTH

-t
Gabriel OrozcoSolution ArchitectCommented:
kgp43:

please do not doubt what you are reading in the logs.

file size is being restricted, so you need to increase it's value.

that's why I comment you about increasing all these values in .htaccess, but looking you are in windows, maybe this setting is not being honored. It should, however.

would you mind to try adding all settings I recommended in a previous post into your php.ini an restart apache and try again?
kgp43Author Commented:
Please note that every upload from the form to the tmp folder is okay, it's just the FTP move that goes wrong.
The max_file_size errors in the logs is becuase they try to upload a file bigger than the allowed 60MB.

Redimido: I can try to change php.ini to the values you suggest, but I though that was only to the form-uploader and not FTP?
decoleurCommented:
how are you instantiating the phpftp function?

how did you change it to accomodate curl?
kgp43Author Commented:
www.savefile.com/filehost/test.php
(please note you wont get any error when a file is 0 byte)

------------------------------------------------------------------------------------------------------------------
this is the code at the test uploader:


<?
// Get time, 36000 seconds (10h) (60MB at 10Kb/sec)
set_time_limit(36000);

// Display all errors
ini_set('display_errors', true);

// Error handling, 0=off 1=on
error_reporting(1);

// Check if any file is submitted
if (!$_FILES['upld_file']['name']) {

      echo "<font color=red>No file selected</font>";
      exit();
}

// Get the file extention
$file_type = $_FILES['upld_file']['name'];
$file_type = pathinfo($file_type);
$file_type = strtolower($file_type['extension']);

// FTP server info
$username = 'user';
$password = 'pass';
$ip = '123.123.123.123';
$path = 'testuploads/';

// FTP url
$ftpURL = 'ftp://'.$ip.'/'.$path;

// File info
$filepath = $_FILES['upld_file']['tmp_name'];
$filename = $_FILES['upld_file']['name'];
$fp = fopen($filepath, 'rb');

// Check for errors
switch ($_FILES['upld_file']['error']) // http://www.php.net/manual/en/features.file-upload.errors.php
{

      case 1:
            echo "<font color=red>The uploaded file exceeds the upload_max_filesize directive in php.ini</font>";
            break;
      
      case 2:
            echo "<font color=red>The uploaded file exceeds the MAX_FILE_SIZE directive that was specified in the HTML form</font>";
            break;
      
      case 3:
            echo "<font color=red>The uploaded file was only partially uploaded</font>";
            break;
      
      case 4:
            echo "<font color=red>No file was uploaded</font>";
            break;
      
      case 6:
            echo "<font color=red>Missing a temporary folder</font>";
            break;
      
      case 0:
            
            // Post filesize
            echo $_FILES['upld_file']['size']." bytes ready to be uploaded (not uploaded yet)<br />
            <br />
            --------------------------------------------<br />
            <b>STATUS:</b><br />
            --------------------------------------------<br />
            <br />";
      
            // ASCII filetypes
            $ascii_filetypes = array('htm', 'html', 'shtml', 'cnf', 'css', 'forward', 'map', 'pwd', 'txt', 'grp', 'ctl', 'nfo');
      
            if (in_array($file_type, $ascii_filetypes)) {
                        
                  $ch = curl_init();
                  curl_setopt($ch, CURLOPT_URL, $ftpURL.$filename);
                  curl_setopt($ch, CURLOPT_USERPWD, "$username:$password");
                  curl_setopt($ch, CURLOPT_INFILE, $fp);
                  curl_setopt($ch, CURLOPT_INFILESIZE, filesize($filepath));
                  curl_setopt($ch, CURLOPT_UPLOAD, 1);
                  curl_exec($ch);
                  fclose($fp);
                  
                  if(curl_errno($ch)) {
                        echo 'Error uploading your file using ASCII mode<br />'.curl_error($ch);
                  }else{
                        curl_close($ch);
                        echo "File uploaded using ASCII mode<br />";
                        echo "Filename: $filename<br />";
                        echo "Filesize: ".$_FILES['upld_file']['size']." bytes<br />";
                  }
            
            }else{
                  
                  $ch = curl_init();
                  curl_setopt($ch, CURLOPT_URL, $ftpURL.$filename);
                  curl_setopt($ch, CURLOPT_USERPWD, "$username:$password");
                  curl_setopt($ch, CURLOPT_INFILE, $fp);
                  curl_setopt($ch, CURLOPT_INFILESIZE, filesize($filepath));
                  curl_setopt($ch, CURLOPT_UPLOAD, 1);
                  curl_exec($ch);
                  fclose($fp);
                  
                  if(curl_errno($ch)) {
                        echo 'Error uploading your file using BINARY mode<br />'.curl_error($ch);
                  }else{
                        curl_close($ch);
                        echo "File uploaded using BINARY mode<br />";
                        echo "Filename: $filename<br />";
                        echo "Filesize: ".$_FILES['upld_file']['size']." bytes<br />";
                  }
                  
            }
      break;
      
}
echo "<br /><br /><br />
<font color=red><b>Remember to paste and copy the entire text on this page when you report back! (including this one)</b></font>";
?>

kgp43Author Commented:
found the error, seems the tmp folder was too small.
Raised it and everything is working again.

what do i do about the points?
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Linux Networking

From novice to tech pro — start learning today.