PHP libssh2 scp_recv transfers incomplete files

I've been using ssh2_scp_recv() to download .tar.gz files.  Each .tar.gz file ranges from 250 KB to 2 MB.  However, I only receive about 140-500 KB of those files, and all 6 I'm downloading are incomplete.

When I used FileZilla/SFTP to download the files, I was able to get the complete file with no problems.

Any ideas?

I am removing the time limit for script execution:
set_time_limit (0);
Joe CoolAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Richard QuadlingSenior Software DeveloperCommented:
Is it possible you could turn on all errors/warnings/notices/etc.

Make sure there are no errors/warnings/notices in your code.

Make sure you have both a compatible library locally for your version of PHP and make sure that there are no compatibility issues with the remote version (I don't know the protocol, but sometimes, different versions of libraries don't like to communicate - can't think this is the issue, unless there is a bug).

Showing my lack of knowledge on the protocol, when you use filezilla/SFTP, is this using the EXACT SAME protocol?

If so, are you able to install Wireshark (a packet sniffer) and see if the packets are being sent/received the same way? Is there something different between the two apps?

A "have you plugged it in question" ... do you have enough disk space?

Same code on a different machine? Still fails?

0
Joe CoolAuthor Commented:
Max error reporting has been enabled, and nothing is reported.  --  error_reporting(E_ALL);

I'm using the latest version of libssh2 (.18) from here.  I compiled and installed it according to the instructions:
http://sourceforge.net/projects/libssh2/

Disk space isn't a problem.

I am going to try running more tests, having it transfer from other servers, and I'll put the code on another server if necessary.  There's just no good reason I can come up with as to why this isn't working.  I hadn't thought to run a packet sniffer, but I suppose it can't hurt.
0
Joe CoolAuthor Commented:
Oh, and here's the code in question, though it's pretty straightforward.
$ssh = ssh2_connect(SSHSERVER, 22);
 
if (ssh2_auth_password($ssh, $login, $password)) {
  echo "PBX Authentication Successful!<br />\n";
} else {
  die('PBX Authentication Failed...<br />\n');
}
 
$olddate = DATELASTRUN;
//set date to previous day so we are sure to get all the CSV files
$olddate = fixdate($olddate, -1);
 
$today = date("y-m-d");
$count = 0;
while ($olddate <= $today) {
	
	//new .tar.gz method to try to speed things up.
	$tarcommand = 'cd '.PBXDIR.'/'.$olddate.' && tar -zcf csv'.$olddate.'.tar.gz *';
	//echo $tarcommand.'<br />';
	$stream = ssh2_exec($ssh, $tarcommand);
	if (!ssh2_scp_recv($ssh, PBXDIR.'/'.$olddate.'/csv'.$olddate.'.tar.gz', $csvdir.'/csv'.$olddate.'.tar.gz')) {
		echo 'csv'.$olddate.'.tar.gz could not be transferred<br />';
	} else {
		echo 'csv'.$olddate.'.tar.gz transferred successfully<br />';
		$count++;
	}
	flush();
	$stream = ssh2_exec($ssh, 'rm -rf '.PBXDIR.'/'.$olddate.'/csv'.$olddate.'.tar.gz');
	
	//$command = 'tar -xvzf '.$csvdir.'/csv'.$olddate.'.tar.gz && rm -rf '.$csvdir.'/csv'.$olddate.'.tar.gz';
	$command = 'tar -xvzf '.$csvdir.'/csv'.$olddate.'.tar.gz ';
	//$command = "pwd";
	//echo $command.'<br />';
	system($command);
	$olddate = fixdate($olddate, 7);
}

Open in new window

0
Cloud Class® Course: Python 3 Fundamentals

This course will teach participants about installing and configuring Python, syntax, importing, statements, types, strings, booleans, files, lists, tuples, comprehensions, functions, and classes.

Richard QuadlingSenior Software DeveloperCommented:
Is it possible that issue the tar.gz command completes, but the actual process is still running on the server?

So you are grabbing the archive as it is being created.

I'm not sure about ssh workings.



Q.

Could you use your server to tell the other server to build the archive and have it send the file to your server?

Hmm. Sounds like a lot of work.


Maybe using a semaphore wrapped around the archive production and a test to see if the semaphore exists will allow you wait until the archive is complete before downloading it.

0
Joe CoolAuthor Commented:
I solved the problem.

It was that the ssh2_exec() tar command was not finishing before the ssh2_scp_recv command began.  So the file would be incompletely downloaded.

The solution was to modify the tar command with the verbose flag and add the following code below the ssh2_exec() command.

stream_set_blocking($stream, true);
while($line = fgets($stream)) {
   flush();
}

Open in new window

0
Richard QuadlingSenior Software DeveloperCommented:
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Vee_ModCommented:
Force accepted.
Vee_Mod
Community Support Moderator
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
PHP

From novice to tech pro — start learning today.