We help IT Professionals succeed at work.

HTTP Post Maximum Characters

abgtemp
abgtemp asked
on
Medium Priority
629 Views
Last Modified: 2012-05-11
I'm trying to use HTTP Post to send the contents of a log file to a PostgreSQL database.  The log I am using for my test is only about 400K in size and has about 5000 lines of text. Only about a quarter of the log is making it to the DB. Is there a setting either in PHP or Apache that controls the amount of data or the limit of characters that can be sent via HTTP Post? I know web browsers have a limitation on them, but I am doing this external from a browser using WinHttp.WinHttpRequest.5.1 on Windows from a vbscript.

Didn't know if Apache also has anything to do with this. My php.ini file has been changed to the following and still no change.

post_max_size = 100M
max_execution_time = 180    
max_input_time = 60
memory_limit = 100M



Comment
Watch Question

Author

Commented:
None of those articles were helpful. They all refer to doing what I have already done with increasing the post_max_size
Most Valuable Expert 2011
Author of the Year 2014

Commented:
about a quarter of the log is making it to the DB

How do you test this?  Are you looking in the DB or are you looking at the raw post data?

Also, have you tried making several smaller posts, perhaps with a key to coordinate the strings?  Then you could reassemble the data on the server with the DB and perform the DB update from the collected posts.

Author

Commented:
I am able to test it by looking at what made it to the Database and what what is the log file.
Most Valuable Expert 2011
Author of the Year 2014

Commented:
Can you get the strlen() value for the raw post data?  I would want to see whether this is exactly the same as the data that was sent, or the same as the data that was put into the data base.  The idea is to try to isolate where the truncation is occurring.

Author

Commented:
I mis-spoked earlier. It's only passing about a tenth of the data. There are 474,047 chars in the log. 47,995 chars of that is being passed to the database via http post.
Most Valuable Expert 2011
Author of the Year 2014

Commented:
I think it's time for us to see the code that is POSTing the data and the code that receives the POSTed data.

Author

Commented:

VBScript that captures the data and performs the HTTP Post
Set objFSO = CreateObject("Scripting.FileSystemObject")
Set objFile = objFSO.GetFile("C:\log.txt")
Set objTextStream = objFile.OpenAsTextStream(1)
logData = objTextStream.ReadAll
objTextStream.Close


'#Remove Any Double or Single Quotes from Log
logData = replace(replace(logData, "'", ""), Chr(34), "")


wscript.sleep 2000


Post_This = "node=ios2101&bkuptype=tsm&date=20110416&startt=1135&endt=1136&type=blog&status=0&log="&logdata



    Set objHTTP = CreateObject("WinHttp.WinHttpRequest.5.1")

	    objHTTP.SetProxy 2,"mywebserver.org:80"
            objHTTP.Open "POST", "http://mywebserver.org/lib/post.php", True
	    objHTTP.SetRequestHeader "User-Agent", "Post Agent"
	    objHTTP.setRequestHeader "Content-Type", "application/x-www-form-urlencoded"
	    objHTTP.send Post_This
	    wscript.sleep 3000
     	    message = VBCrlf & "Response from Server: "&VBCrlf& objHTTP.ResponseText
            wscript.echo message
    Set objHTTP = Nothing




wscript.sleep 500

wscript.echo "Done"

Open in new window




PHP script that receives the data and selects the type of handler
<?php
	if ($_SERVER['HTTP_USER_AGENT'] != 'Post Agent') {
		echo "\nfailure: must post from a valid server\n";
		exit;
	}
	
	define("VALID_POST", 1);
	define("IN_TASKSHEET", 1);
	
	require_once('db.php');
	require_once('interface.php');
	
	$db = new mydb;
	
	$keys = array_keys($_POST);
	
	foreach ($keys as $key) {
		$_POST[$key] = $db->escape($_POST[$key]);
	}
	
	extract($_POST);
	
	$rc = 'failure';
	
	if (file_exists('post/' . $type . '.post.php')) {
		print_r($_POST);
		echo "\n\n";
		
		require_once('post/' . $type . '.post.php');
	} else {
		echo "\nfailure: type handler does not exist\n";
		exit;
	}

	echo "\n" . $rc . "\n";
?>

Open in new window



Handler that perform the input into the table
 
<?php if (!defined("VALID_POST")) { echo 'failure: you can\'t post to this url directly'; exit; }
	$rc = 'success';
	
	$sql = 'INSERT INTO "mylogs" ("id", "log") VALUES (next_id(\'mylogs\'), \'' . $log . '\');';
	
	if ($db->query($sql) === true)	
		echo "posted data\n";
	else {
		echo "Could not post Data\nfailure\n";
		exit;
	}
	
	$rc = 'success';
?>

Open in new window

Most Valuable Expert 2011
Author of the Year 2014
Commented:
Unlock this solution and get a sample of our free trial.
(No credit card required)
UNLOCK SOLUTION

Author

Commented:
I added the var_dump($_POST) and the length of the string was response was 48,303. I was looking to see if there was a limit on a variable in PHP, but it doesn't look there is one.
Dave BaldwinFixer of Problems
CERTIFIED EXPERT
Most Valuable Expert 2014

Commented:
Any chance your log file has non-ASCII characters?  http://msdn.microsoft.com/en-us/library/hwfw5c59%28v=VS.85%29.aspx
Most Valuable Expert 2011
Author of the Year 2014

Commented:
There is no limit to the length of a string in PHP.

Can you please give us both of the string lengths? Thanks.
Most Valuable Expert 2011
Author of the Year 2014

Commented:
Also, could you please give me a URL for the log file so I can look at the 400K string?  Thanks.
Most Valuable Expert 2011
Author of the Year 2014

Commented:
This appears to say what we already believe...
http://stackoverflow.com/questions/2364840/what-is-the-size-limit-of-a-post-request

Can you install this script and post a link to the URL here, please?

Can you tell us if you are using Suhosin?
<?php phpinfo();

Open in new window

Unlock the solution to this question.
Thanks for using Experts Exchange.

Please provide your email to receive a sample view!

*This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

OR

Please enter a first name

Please enter a last name

8+ characters (letters, numbers, and a symbol)

By clicking, you agree to the Terms of Use and Privacy Policy.