Link to home
Start Free TrialLog in
Avatar of abgtemp
abgtempFlag for United States of America

asked on

HTTP Post Maximum Characters

I'm trying to use HTTP Post to send the contents of a log file to a PostgreSQL database.  The log I am using for my test is only about 400K in size and has about 5000 lines of text. Only about a quarter of the log is making it to the DB. Is there a setting either in PHP or Apache that controls the amount of data or the limit of characters that can be sent via HTTP Post? I know web browsers have a limitation on them, but I am doing this external from a browser using WinHttp.WinHttpRequest.5.1 on Windows from a vbscript.

Didn't know if Apache also has anything to do with this. My php.ini file has been changed to the following and still no change.

post_max_size = 100M
max_execution_time = 180    
max_input_time = 60
memory_limit = 100M



Avatar of abgtemp

ASKER

None of those articles were helpful. They all refer to doing what I have already done with increasing the post_max_size
about a quarter of the log is making it to the DB

How do you test this?  Are you looking in the DB or are you looking at the raw post data?

Also, have you tried making several smaller posts, perhaps with a key to coordinate the strings?  Then you could reassemble the data on the server with the DB and perform the DB update from the collected posts.
Avatar of abgtemp

ASKER

I am able to test it by looking at what made it to the Database and what what is the log file.
Can you get the strlen() value for the raw post data?  I would want to see whether this is exactly the same as the data that was sent, or the same as the data that was put into the data base.  The idea is to try to isolate where the truncation is occurring.
Avatar of abgtemp

ASKER

I mis-spoked earlier. It's only passing about a tenth of the data. There are 474,047 chars in the log. 47,995 chars of that is being passed to the database via http post.
I think it's time for us to see the code that is POSTing the data and the code that receives the POSTed data.
Avatar of abgtemp

ASKER


VBScript that captures the data and performs the HTTP Post
Set objFSO = CreateObject("Scripting.FileSystemObject")
Set objFile = objFSO.GetFile("C:\log.txt")
Set objTextStream = objFile.OpenAsTextStream(1)
logData = objTextStream.ReadAll
objTextStream.Close


'#Remove Any Double or Single Quotes from Log
logData = replace(replace(logData, "'", ""), Chr(34), "")


wscript.sleep 2000


Post_This = "node=ios2101&bkuptype=tsm&date=20110416&startt=1135&endt=1136&type=blog&status=0&log="&logdata



    Set objHTTP = CreateObject("WinHttp.WinHttpRequest.5.1")

	    objHTTP.SetProxy 2,"mywebserver.org:80"
            objHTTP.Open "POST", "http://mywebserver.org/lib/post.php", True
	    objHTTP.SetRequestHeader "User-Agent", "Post Agent"
	    objHTTP.setRequestHeader "Content-Type", "application/x-www-form-urlencoded"
	    objHTTP.send Post_This
	    wscript.sleep 3000
     	    message = VBCrlf & "Response from Server: "&VBCrlf& objHTTP.ResponseText
            wscript.echo message
    Set objHTTP = Nothing




wscript.sleep 500

wscript.echo "Done"

Open in new window




PHP script that receives the data and selects the type of handler
<?php
	if ($_SERVER['HTTP_USER_AGENT'] != 'Post Agent') {
		echo "\nfailure: must post from a valid server\n";
		exit;
	}
	
	define("VALID_POST", 1);
	define("IN_TASKSHEET", 1);
	
	require_once('db.php');
	require_once('interface.php');
	
	$db = new mydb;
	
	$keys = array_keys($_POST);
	
	foreach ($keys as $key) {
		$_POST[$key] = $db->escape($_POST[$key]);
	}
	
	extract($_POST);
	
	$rc = 'failure';
	
	if (file_exists('post/' . $type . '.post.php')) {
		print_r($_POST);
		echo "\n\n";
		
		require_once('post/' . $type . '.post.php');
	} else {
		echo "\nfailure: type handler does not exist\n";
		exit;
	}

	echo "\n" . $rc . "\n";
?>

Open in new window



Handler that perform the input into the table
 
<?php if (!defined("VALID_POST")) { echo 'failure: you can\'t post to this url directly'; exit; }
	$rc = 'success';
	
	$sql = 'INSERT INTO "mylogs" ("id", "log") VALUES (next_id(\'mylogs\'), \'' . $log . '\');';
	
	if ($db->query($sql) === true)	
		echo "posted data\n";
	else {
		echo "Could not post Data\nfailure\n";
		exit;
	}
	
	$rc = 'success';
?>

Open in new window

ASKER CERTIFIED SOLUTION
Avatar of Ray Paseur
Ray Paseur
Flag of United States of America image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Avatar of abgtemp

ASKER

I added the var_dump($_POST) and the length of the string was response was 48,303. I was looking to see if there was a limit on a variable in PHP, but it doesn't look there is one.
Avatar of Dave Baldwin
Any chance your log file has non-ASCII characters?  http://msdn.microsoft.com/en-us/library/hwfw5c59%28v=VS.85%29.aspx
There is no limit to the length of a string in PHP.

Can you please give us both of the string lengths? Thanks.
Also, could you please give me a URL for the log file so I can look at the 400K string?  Thanks.
This appears to say what we already believe...
http://stackoverflow.com/questions/2364840/what-is-the-size-limit-of-a-post-request

Can you install this script and post a link to the URL here, please?

Can you tell us if you are using Suhosin?
<?php phpinfo();

Open in new window