?
Solved

HTTP Post Maximum Characters

Posted on 2011-04-22
14
Medium Priority
?
497 Views
Last Modified: 2012-05-11
I'm trying to use HTTP Post to send the contents of a log file to a PostgreSQL database.  The log I am using for my test is only about 400K in size and has about 5000 lines of text. Only about a quarter of the log is making it to the DB. Is there a setting either in PHP or Apache that controls the amount of data or the limit of characters that can be sent via HTTP Post? I know web browsers have a limitation on them, but I am doing this external from a browser using WinHttp.WinHttpRequest.5.1 on Windows from a vbscript.

Didn't know if Apache also has anything to do with this. My php.ini file has been changed to the following and still no change.

post_max_size = 100M
max_execution_time = 180    
max_input_time = 60
memory_limit = 100M



0
Comment
Question by:abgtemp
14 Comments
 

Author Comment

by:abgtemp
ID: 35450308
None of those articles were helpful. They all refer to doing what I have already done with increasing the post_max_size
0
 
LVL 111

Expert Comment

by:Ray Paseur
ID: 35450842
about a quarter of the log is making it to the DB

How do you test this?  Are you looking in the DB or are you looking at the raw post data?

Also, have you tried making several smaller posts, perhaps with a key to coordinate the strings?  Then you could reassemble the data on the server with the DB and perform the DB update from the collected posts.
0
Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 

Author Comment

by:abgtemp
ID: 35450951
I am able to test it by looking at what made it to the Database and what what is the log file.
0
 
LVL 111

Expert Comment

by:Ray Paseur
ID: 35451051
Can you get the strlen() value for the raw post data?  I would want to see whether this is exactly the same as the data that was sent, or the same as the data that was put into the data base.  The idea is to try to isolate where the truncation is occurring.
0
 

Author Comment

by:abgtemp
ID: 35451098
I mis-spoked earlier. It's only passing about a tenth of the data. There are 474,047 chars in the log. 47,995 chars of that is being passed to the database via http post.
0
 
LVL 111

Expert Comment

by:Ray Paseur
ID: 35451151
I think it's time for us to see the code that is POSTing the data and the code that receives the POSTed data.
0
 

Author Comment

by:abgtemp
ID: 35451186

VBScript that captures the data and performs the HTTP Post
Set objFSO = CreateObject("Scripting.FileSystemObject")
Set objFile = objFSO.GetFile("C:\log.txt")
Set objTextStream = objFile.OpenAsTextStream(1)
logData = objTextStream.ReadAll
objTextStream.Close


'#Remove Any Double or Single Quotes from Log
logData = replace(replace(logData, "'", ""), Chr(34), "")


wscript.sleep 2000


Post_This = "node=ios2101&bkuptype=tsm&date=20110416&startt=1135&endt=1136&type=blog&status=0&log="&logdata



    Set objHTTP = CreateObject("WinHttp.WinHttpRequest.5.1")

	    objHTTP.SetProxy 2,"mywebserver.org:80"
            objHTTP.Open "POST", "http://mywebserver.org/lib/post.php", True
	    objHTTP.SetRequestHeader "User-Agent", "Post Agent"
	    objHTTP.setRequestHeader "Content-Type", "application/x-www-form-urlencoded"
	    objHTTP.send Post_This
	    wscript.sleep 3000
     	    message = VBCrlf & "Response from Server: "&VBCrlf& objHTTP.ResponseText
            wscript.echo message
    Set objHTTP = Nothing




wscript.sleep 500

wscript.echo "Done"

Open in new window




PHP script that receives the data and selects the type of handler
<?php
	if ($_SERVER['HTTP_USER_AGENT'] != 'Post Agent') {
		echo "\nfailure: must post from a valid server\n";
		exit;
	}
	
	define("VALID_POST", 1);
	define("IN_TASKSHEET", 1);
	
	require_once('db.php');
	require_once('interface.php');
	
	$db = new mydb;
	
	$keys = array_keys($_POST);
	
	foreach ($keys as $key) {
		$_POST[$key] = $db->escape($_POST[$key]);
	}
	
	extract($_POST);
	
	$rc = 'failure';
	
	if (file_exists('post/' . $type . '.post.php')) {
		print_r($_POST);
		echo "\n\n";
		
		require_once('post/' . $type . '.post.php');
	} else {
		echo "\nfailure: type handler does not exist\n";
		exit;
	}

	echo "\n" . $rc . "\n";
?>

Open in new window



Handler that perform the input into the table
 
<?php if (!defined("VALID_POST")) { echo 'failure: you can\'t post to this url directly'; exit; }
	$rc = 'success';
	
	$sql = 'INSERT INTO "mylogs" ("id", "log") VALUES (next_id(\'mylogs\'), \'' . $log . '\');';
	
	if ($db->query($sql) === true)	
		echo "posted data\n";
	else {
		echo "Could not post Data\nfailure\n";
		exit;
	}
	
	$rc = 'success';
?>

Open in new window

0
 
LVL 111

Accepted Solution

by:
Ray Paseur earned 1200 total points
ID: 35451227
OK, that looks good.  Try using var_dump($_POST) before you make any modification to $_POST and also printing out the strlen($log) right before you construct the $sql statement
0
 

Author Comment

by:abgtemp
ID: 35451287
I added the var_dump($_POST) and the length of the string was response was 48,303. I was looking to see if there was a limit on a variable in PHP, but it doesn't look there is one.
0
 
LVL 84

Expert Comment

by:Dave Baldwin
ID: 35452332
Any chance your log file has non-ASCII characters?  http://msdn.microsoft.com/en-us/library/hwfw5c59%28v=VS.85%29.aspx
0
 
LVL 111

Expert Comment

by:Ray Paseur
ID: 35453145
There is no limit to the length of a string in PHP.

Can you please give us both of the string lengths? Thanks.
0
 
LVL 111

Expert Comment

by:Ray Paseur
ID: 35453154
Also, could you please give me a URL for the log file so I can look at the 400K string?  Thanks.
0
 
LVL 111

Expert Comment

by:Ray Paseur
ID: 35453175
This appears to say what we already believe...
http://stackoverflow.com/questions/2364840/what-is-the-size-limit-of-a-post-request

Can you install this script and post a link to the URL here, please?

Can you tell us if you are using Suhosin?
<?php phpinfo();

Open in new window

0

Featured Post

Upgrade your Question Security!

Your question, your audience. Choose who sees your identity—and your question—with question security.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Nothing in an HTTP request can be trusted, including HTTP headers and form data.  A form token is a tool that can be used to guard against request forgeries (CSRF).  This article shows an improved approach to form tokens, making it more difficult to…
This article discusses four methods for overlaying images in a container on a web page
Explain concepts important to validation of email addresses with regular expressions. Applies to most languages/tools that uses regular expressions. Consider email address RFCs: Look at HTML5 form input element (with type=email) regex pattern: T…
This tutorial will teach you the core code needed to finalize the addition of a watermark to your image. The viewer will use a small PHP class to learn and create a watermark.
Suggested Courses

621 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question