send cURL post to be received exactly like form post

Posted on 2010-08-17
Last Modified: 2013-11-22
I am trying to send simple entries in a form using PHP cURL so a remote server that the entries go to receives them in exactly the same manner as if sent from a plain and ordinary form.  So far, the remote server accepts post directly from the form but not when sent by this PHP attempt.  fopen and fsockopen etc. are set to "off" by the host (Yahoo) that I use, so cURL seems to be the best alternative.

As far as I can tell, no matter how the remote server is set up, if an ordinary post transmission is duplicated in its entirety, the server should accept the entries using cURL if it accepts the posted entries directly from the form.
    $header[]='Content-type: application/x-www-form-urlencoded';
    $useragent = 'Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv: Gecko/20100722 Toolbar 2.0.1 Firefox/3.6.8 ( .NET CLR 3.5.30729)   AutoPager/';

    $ch = curl_init();
    curl_setopt($ch, CURLOPT_HTTPHEADER, $header);
    curl_setopt($ch, CURLOPT_URL, $URL);
    curl_setopt($ch, CURLOPT_USERAGENT, $useragent);
    curl_setopt($ch, CURLOPT_REFERER, $referer);
    curl_setopt($ch, CURLOPT_POST, true);
    curl_setopt($ch, CURLOPT_POSTFIELDS, $_POST);
    curl_exec ($ch);    
    curl_close ($ch);

Open in new window

The remote server will not accept the entries using this code.

From Firebug - Response Headers when form entries are posted:

Date: Tue, 17 Aug 2010 22:36:34 GMT
Server: Apache/2.2.8 (Fedora)
Connection: close
Transfer-Encoding: chunked
Content-Type: text/html

From Firebug - Request Headers when form entries are posted:

User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv: Gecko/20100722 Toolbar 2.0.1 Firefox/3.6.8 ( .NET CLR 3.5.30729) AutoPager/
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 115
Connection: keep-alive
Referer: http://mysite-form.htm

What can be done to make the entries be received the same as if sent by the form?
     FORM action="http://remote_server.cgi"  method="POST"

Open in new window

Question by:mainiak
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 7
  • 4
  • 4

Accepted Solution

merwetta1 earned 250 total points
ID: 33460879
I compared your curl code to some I have working on a site, and posted a hybrid below. If that works, you could try adding back in some of the other header commands.
$fields = "";
foreach($_POST as $key => $value ) $fields .= "$key=" . urlencode( $value ) . "&";

$ch = curl_init($URL); 
curl_setopt($ch, CURLOPT_HEADER, 0); // set to 0 to eliminate header info from response
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); // Returns response data instead of TRUE(1)
curl_setopt($ch, CURLOPT_POSTFIELDS, rtrim( $fields, "& " )); // use HTTP POST to send form data
$resp = curl_exec($ch); //execute post and get results
curl_close ($ch);

Open in new window


Author Comment

ID: 33461114
Thanks for the reply, merwetta1.

I copied and pasted, inserted the actual URL, and upon submit, I get a blank page but no errors show in scripts.log.

Expert Comment

ID: 33461969
The code I provided doesn't have any output. I'm assuming you have some other code that executes after the curl, or perhaps I don't fully understand what you are trying to accomplish. If your CGI script outputs a result string, it will be captured in the $resp variable. So you could add one more line:
echo $resp;

What does your CGI script do and can you tell if it's running properly now?
Visualize your virtual and backup environments

Create well-organized and polished visualizations of your virtual and backup environments when planning VMware vSphere, Microsoft Hyper-V or Veeam deployments. It helps you to gain better visibility and valuable business insights.

LVL 110

Expert Comment

by:Ray Paseur
ID: 33464738
What you're trying to do here is commonly used to hack or attack web sites.  If the remote site wants you to make automated posts, the owner of the site should give you clear instructions showing how to use their API.  

In addition, this line is almost certainly wrong - cURL POSTFIELDS are a string -- not the same format as the array $_POST.
    curl_setopt($ch, CURLOPT_POSTFIELDS, $_POST);

Please post the ACTUAL URL of the page you're trying to target with automated post information.  Without seeing that, we have almost no hope of helping you.

Thanks, ~Ray

Author Comment

ID: 33467725
Thank you both for your replies.

These are not automated posts and it is the vendor's remote server which has the cgi script.

Users make entries into a form on my site which are then sent to a vendor's remote server which generates a variable length page of results for the users' perusal.  I want the results to be displayed on my site but I discover that there is a problem with iframe height so I am trying PHP in a separate file to avoid an iframe.  Then maybe I can cache the result pages and display them without needing an iframe.  The Javascript remote server security issue looks daunting so I am trying to use a PHP solution first.

The URL that entries go to is:

You can see the output from the form by going to:

If you scroll down, you will see that the quote results are in an iframe of fixed height which is not the best because the result pages are of varying height.  The alternative is a double scroll bar which is inconvenient for any user.

I have been unable to find any better alternative to the iframe height problem and had hoped that this would be simpler than it is.
LVL 110

Assisted Solution

by:Ray Paseur
Ray Paseur earned 250 total points
ID: 33468121
Not sure what to tell you about iFrames, but this script will POST the data to the remote URL and retrieve the HTML generated by the server.  HTH, ~Ray
<?php // RAY_temp_curl_post_example.php

function curl_post($url, $post_array, $timeout=2, $error_report=FALSE)
    $post_string = '';
    foreach ($post_array as $key => $val)
        $post_string .= urlencode($key) . '=' . urlencode($val) . '&';
    $post_string = rtrim($post_string, '&');

    $curl = curl_init();
    curl_setopt($curl, CURLOPT_URL,            $url);
    curl_setopt($curl, CURLOPT_HEADER,         FALSE);
    curl_setopt($curl, CURLOPT_POST,           TRUE);
    curl_setopt($curl, CURLOPT_POSTFIELDS,     $post_string);
    curl_setopt($curl, CURLOPT_TIMEOUT,        $timeout);
    curl_setopt($curl, CURLOPT_RETURNTRANSFER, TRUE);

    $htm = curl_exec($curl);
    $err = curl_errno($curl);
    $inf = curl_getinfo($curl);

    if (!$htm)
        if ($error_report)
            echo "CURL FAIL: $url TIMEOUT=$timeout, CURL_ERRNO=$err";
            echo "<pre>\n";
            echo "</pre>\n";
        return FALSE;

    return $htm;

<INPUT type="hidden" name="UserLocation" value="1447">
<INPUT TYPE="hidden" name="SortOverride1" value="A">
<SELECT name="State" size="1"><OPTION selected value="5">California</OPTION></SELECT>
<SELECT NAME="BirthMonth"><OPTION selected value="6">June</OPTION></SELECT>
<SELECT NAME="Birthday" ><OPTION selected>15</OPTION></SELECT>
<SELECT NAME="BirthYear" ><OPTION selected>1970</OPTION></SELECT>
Male&nbsp;<INPUT type="radio" checked name="Sex" value="M">
No&nbsp;<INPUT type="radio" checked name="Smoker" value="N">
<SELECT name="Health" size="1" ><OPTION selected value="PP">Preferred Plus</OPTION></SELECT>
<SELECT name="NewCategory"><OPTION SELECTED value="5">20 Year Level Term</OPTION></SELECT>&nbsp;
<SELECT name="FaceAmount" size="1"><OPTION selected value="500000">$500,000</OPTION></SELECT>
<SELECT name="ModeUsed" size="1"><OPTION selected value="M"> Monthly </OPTION></SELECT>
<SELECT NAME="CompRating"><OPTION selected value="4">A- &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Excellent</OPTION></SELECT>
<INPUT type="submit" name="CqsComparison" value=" Compare Now ">

$args["UserLocation"] = "1447";
$args["SortOverride1"] = "A";
$args["State"] = "5";
$args["BirthMonth"] = "6";
$args["Birthday"] = "15";
$args["BirthYear"] = "1970";
$args["Sex"] = "M";
$args["Smoker"] = "N";
$args["Health"] = "PP";
$args["NewCategory"] = "5";
$args["FaceAmount"] = "500000";
$args["ModelUsed"] = "M";
$args["CompRating"] = "4";
$args["CqsComparison"] = " Compare Now ";

$url = "";

$htm = curl_post($url, $args, 6, TRUE);
if (!$htm) die("NO $url");

echo "<pre>";
echo htmlentities($htm);

Open in new window


Author Comment

ID: 33469296
Thanks Ray, but there is still the basic problem.

The hidden values I can change.  All other values are entered by each visitor. The values on the page when you open it are just default values which the life insurance shopper changes for their own situation.

The vendor sets the file that way when they send it to their customers (like me), but I may change the default to blanks if I conclude that buyers may prefer that.

Listing the values of default entries in the code will not suffice.  Every submission is different as you would see if you ran your own quote.

$_POST varies with every visitor and that is what I am unable to get sent in an acceptable form by PHP.
LVL 110

Expert Comment

by:Ray Paseur
ID: 33469393
This is your application, not mine, and I don't have time to write all the code for you.  Can you please break it down a little bit into questions we can answer?  For example, are you unable to put a form on the web page, receive the input and copy the values out of $_POST?  Where is the sticking point?

Expert Comment

ID: 33470835
mainiak, Ray's solution rocks. I think the only thing you aren't getting is that he used the $args array to reproduce a sample $_POST array. As long as your form has all the appropriately named fields, you could replace line 89 of Ray's script with this and it should work:

$htm = curl_post($url, $_POST, 6, TRUE);

Then you can delete all the lines that start with "$args".

Author Comment

ID: 33471982
Please let there be no misunderstanding. I am well aware that this is my application and nowhere do I ask anyone to "write all the code".

I have written code and am trying to understand its deficiencies.  I have previously outlined the situation, but will explain again in different words so as to clarify the "sticking point".

I am devising a substitute to posting from a form directly to a remote server.  A PHP file is now to be in the middle:           Form - - - PHP - - - remote server.

Of itself, the form has been working for many years and continues to do so, but for reasons already explained, I now want entries to go from the form to a PHP file and then be passed on to the remote server.

Here is the action in the form as provided by the vendor:

<FORM action="" method="POST">

I changed that to:

<FORM action="" method="POST">

So the POST goes first to my file in which there is PHP code.

I want the various posted entries entered by visitors to be passed along by PHP to the vendor just the same as if by the form directly.

That is what is meant by "... so a remote server that the entries go to receives them in exactly the same manner as if sent from a plain and ordinary form... " 

The code I posted, and the many other variations I have tried, results in a message from the remote server "Error in form found... ", instead of generating quote results, so my attempt does not duplicate transmission directly from the usual and common type of form.

You said earlier "...  this line is almost certainly wrong - cURL POSTFIELDS are a string -- not the same format as the array $_POST.
    curl_setopt($ch, CURLOPT_POSTFIELDS, $_POST);... "

At I find:
CURLOPT_POSTFIELDS         The full data to post in a HTTP "POST" operation. To post a file, prepend a filename with @ and use the full path. This can either be passed as a urlencoded string like 'para1=val1&para2=val2&...' or as an array with the field name as key and field data as value. If value is an array, the Content-Type header will be set to multipart/form-data.

If CURLOPT_POSTFIELDS is not appropriate, what is?  I am unable to find a suitable substitute.

I assumed that POST from the form, being an array, would be passed along as such by
              curl_setopt($ch, CURLOPT_POSTFIELDS, $_POST);

If I am mistaken, what is the remedy?

Based on the statement, I removed  $header[]='Content-type: application/x-www-form-urlencoded'; and substituted $header[]='Content-type: text/html'; based on the Request and Response headers, but the "Error in form... " remains.

  So far the vendor is unable to help with this problem.

In short, what does it take for the remote server to receive differing sets of POST form entries through the auspices of a PHP file using cURL exactly as if direct from a form?
LVL 110

Expert Comment

by:Ray Paseur
ID: 33478009
Please refer to my sample script posted at ID:33468121.  Lines 6 through 12 convert a post array into the raw post string that is required by this call to CURL.  I have rarely gotten an array to work correctly, so I prefer to use the urlencoded string.

Your general design pattern would work like this:

1. Put up a form on your site to receive the client data in the $_POST array
2. Merge the client input with the hidden input.  You can write this information into an associative array
3. Use the curl_post function shown above to pass the merged data to the remote service and read the response.

However a better solution would be a formal API, with documentation.  As I wrote above, "If the remote site wants you to make automated posts, the owner of the site should give you clear instructions showing how to use their API."  There could be any number of reasons why the remote server is rejecting your data.  Finding and debugging this without help from the owners of the site is a daunting and thankless task.

Author Comment

ID: 33527048
This is now almost there:

The code from merwetta1 (ID: 334760879) is accepted by the remote server provided that the CURLOPT_RETURNTRANSFER line is removed (CURLOPT_HEADER line not needed).  

However, the results page opens in the PHP file (compulife-action.php) but must open at the remote server so relative links on the results page can function.  The vendor refuses to change the links.

If the results page opens at the remote server ($URL), this would then be resolved.    

$fields = "";
foreach($_POST as $key => $value ) $fields .= "$key=" . urlencode( $value ) . "&";  

$ch = curl_init($URL);
curl_setopt($ch, CURLOPT_POSTFIELDS, rtrim( $fields, "& " )); // use HTTP POST to send form data
curl_exec ($ch);  
curl_close ($ch);

Ray, your comments in ID: 33478009 indicates a misunderstanding, e.g., "There could be any number of reasons why the remote server is rejecting your data."   That is not correct.  There is only one reason which is that POST from the vendor's form on my site was not being replicated.  The data is choices made by visitors from a range of options provided by the vendor who wrote the form.

A "formal API, with documentation"?  No, not needed.

"Finding and debugging this without help from the owners of the site is a daunting and thankless task."

Not correct.  No debugging is needed.  The solution is to just replicate POST from a form, as I said in my original question ID: 26410312.

The vendor's remote server has NO requirements except POST from the vendor supplied form.

Using your code in ID:33468121, upon submit, the remote server generates only a 'view source' of results with static entry values, and opens at the PHP file.

I amended your code according to merwetta1 in ID: 33470835.  That also results in a 'view source' of the results page containing whatever values are entered into the form and similarly opens at the PHP file.

However, if in addition, SHOW WHAT CAME BACK is amended to:
//echo "<pre>";
//echo htmlentities($htm);
echo ($htm);
that opens the results page itself, at the PHP file (compulife-action.php), but it needs to open at the remote server.  Doing that resolves this question.  

I have tried everything I can to tell the remote server to simply open the results there, like,
header ("Location:");

etc., but it's all after the fact.  As far as I can tell, it has to be included in the POST but I can't find or think of what is appropriate to that.    


Expert Comment

ID: 33536368
If the only remaining issue is relative links in the HTML, you could do some string replacements to insert the correct "" before any relative links. For example, replace "src=images/" with "src=". (Assuming you have permission to link to the site's graphics of course.)

You have been offered a lot of help and code here. Your original question was: "What can be done to make the entries be received the same as if sent by the form?" I think that has been answered. You should consider awarding points and starting a new question that includes the code of where you are at now.

Author Comment

ID: 33537299
Sure, it has been mostly answered and I appreciate the help because I could not have gotten this far by myself.

I will post a new question as suggested.

The remaining issue is not relative links but that it must open at the remote server like it does direct from the form.  It opens at my php file and functionality is lost because of the relative links in a page that is impossible for me to change.

The results page that contains the relative links is controlled solely by the vendor.  Linking to the site's 100+ graphics is not possible.  They have to be changed at their remote server and the other party has reasons to keep them the way they are.

Author Closing Comment

ID: 33537324
Very helpful, a good start to solving this issue.

Featured Post

SharePoint Admin?

Enable Your Employees To Focus On The Core With Intuitive Onscreen Guidance That is With You At The Moment of Need.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
Echo'd value as selected in dropdown 20 39
Change background images after 5 seconds. 12 42
Best hashing method? 1 16
Load string Array from file 23 37
A Change in PHP Behavior with Session Write Short Circuit ( (Winter 2014)** With the release of PHP 5.6 the session handler changed in a way that many think should be considered a bug.  See the note …
As a financial services provider, your business is impacted by two of the strictest federal regulations on record: the Sarbanes-Oxley Act and the Gramm-Leach-Bliley Act. Correctly implementing faxing into your organization to provide secure, real-ti…
This tutorial will teach you the core code needed to finalize the addition of a watermark to your image. The viewer will use a small PHP class to learn and create a watermark.
The viewer will learn how to create a basic form using some HTML5 and PHP for later processing. Set up your basic HTML file. Open your form tag and set the method and action attributes.: (CODE) Set up your first few inputs one for the name and …

738 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question