abgtemp
asked on
HTTP Post Maximum Characters
I'm trying to use HTTP Post to send the contents of a log file to a PostgreSQL database. The log I am using for my test is only about 400K in size and has about 5000 lines of text. Only about a quarter of the log is making it to the DB. Is there a setting either in PHP or Apache that controls the amount of data or the limit of characters that can be sent via HTTP Post? I know web browsers have a limitation on them, but I am doing this external from a browser using WinHttp.WinHttpRequest.5.1 on Windows from a vbscript.
Didn't know if Apache also has anything to do with this. My php.ini file has been changed to the following and still no change.
post_max_size = 100M
max_execution_time = 180
max_input_time = 60
memory_limit = 100M
Didn't know if Apache also has anything to do with this. My php.ini file has been changed to the following and still no change.
post_max_size = 100M
max_execution_time = 180
max_input_time = 60
memory_limit = 100M
ASKER
None of those articles were helpful. They all refer to doing what I have already done with increasing the post_max_size
about a quarter of the log is making it to the DB
How do you test this? Are you looking in the DB or are you looking at the raw post data?
Also, have you tried making several smaller posts, perhaps with a key to coordinate the strings? Then you could reassemble the data on the server with the DB and perform the DB update from the collected posts.
How do you test this? Are you looking in the DB or are you looking at the raw post data?
Also, have you tried making several smaller posts, perhaps with a key to coordinate the strings? Then you could reassemble the data on the server with the DB and perform the DB update from the collected posts.
ASKER
I am able to test it by looking at what made it to the Database and what what is the log file.
Can you get the strlen() value for the raw post data? I would want to see whether this is exactly the same as the data that was sent, or the same as the data that was put into the data base. The idea is to try to isolate where the truncation is occurring.
ASKER
I mis-spoked earlier. It's only passing about a tenth of the data. There are 474,047 chars in the log. 47,995 chars of that is being passed to the database via http post.
I think it's time for us to see the code that is POSTing the data and the code that receives the POSTed data.
ASKER
VBScript that captures the data and performs the HTTP Post
Set objFSO = CreateObject("Scripting.FileSystemObject")
Set objFile = objFSO.GetFile("C:\log.txt")
Set objTextStream = objFile.OpenAsTextStream(1)
logData = objTextStream.ReadAll
objTextStream.Close
'#Remove Any Double or Single Quotes from Log
logData = replace(replace(logData, "'", ""), Chr(34), "")
wscript.sleep 2000
Post_This = "node=ios2101&bkuptype=tsm&date=20110416&startt=1135&endt=1136&type=blog&status=0&log="&logdata
Set objHTTP = CreateObject("WinHttp.WinHttpRequest.5.1")
objHTTP.SetProxy 2,"mywebserver.org:80"
objHTTP.Open "POST", "http://mywebserver.org/lib/post.php", True
objHTTP.SetRequestHeader "User-Agent", "Post Agent"
objHTTP.setRequestHeader "Content-Type", "application/x-www-form-urlencoded"
objHTTP.send Post_This
wscript.sleep 3000
message = VBCrlf & "Response from Server: "&VBCrlf& objHTTP.ResponseText
wscript.echo message
Set objHTTP = Nothing
wscript.sleep 500
wscript.echo "Done"
PHP script that receives the data and selects the type of handler
<?php
if ($_SERVER['HTTP_USER_AGENT'] != 'Post Agent') {
echo "\nfailure: must post from a valid server\n";
exit;
}
define("VALID_POST", 1);
define("IN_TASKSHEET", 1);
require_once('db.php');
require_once('interface.php');
$db = new mydb;
$keys = array_keys($_POST);
foreach ($keys as $key) {
$_POST[$key] = $db->escape($_POST[$key]);
}
extract($_POST);
$rc = 'failure';
if (file_exists('post/' . $type . '.post.php')) {
print_r($_POST);
echo "\n\n";
require_once('post/' . $type . '.post.php');
} else {
echo "\nfailure: type handler does not exist\n";
exit;
}
echo "\n" . $rc . "\n";
?>
Handler that perform the input into the table
<?php if (!defined("VALID_POST")) { echo 'failure: you can\'t post to this url directly'; exit; }
$rc = 'success';
$sql = 'INSERT INTO "mylogs" ("id", "log") VALUES (next_id(\'mylogs\'), \'' . $log . '\');';
if ($db->query($sql) === true)
echo "posted data\n";
else {
echo "Could not post Data\nfailure\n";
exit;
}
$rc = 'success';
?>
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
I added the var_dump($_POST) and the length of the string was response was 48,303. I was looking to see if there was a limit on a variable in PHP, but it doesn't look there is one.
Any chance your log file has non-ASCII characters? http://msdn.microsoft.com/en-us/library/hwfw5c59%28v=VS.85%29.aspx
There is no limit to the length of a string in PHP.
Can you please give us both of the string lengths? Thanks.
Can you please give us both of the string lengths? Thanks.
Also, could you please give me a URL for the log file so I can look at the 400K string? Thanks.
This appears to say what we already believe...
http://stackoverflow.com/questions/2364840/what-is-the-size-limit-of-a-post-request
Can you install this script and post a link to the URL here, please?
Can you tell us if you are using Suhosin?
http://stackoverflow.com/questions/2364840/what-is-the-size-limit-of-a-post-request
Can you install this script and post a link to the URL here, please?
Can you tell us if you are using Suhosin?
<?php phpinfo();
http://bytes.com/topic/php/answers/3607-get-post-size-data-limit
http://stackoverflow.com/questions/2364840/what-is-the-size-limit-of-a-post-request
http://forums.digitalpoint.com/showthread.php?t=1513000