Want to win a PS4? Go Premium and enter to win our High-Tech Treats giveaway. Enter to Win

x
?
Solved

Http resend script question

Posted on 2004-09-27
32
Medium Priority
?
325 Views
Last Modified: 2008-02-01
Hello!
I would like to write a script that resends all http requests to a single, predefined hostname with the exact same same parameters that the script receives. The script must handle HEAD, GET and POST requests. For file uploads, the script could receive the incoming file, resend the incoming file to the predefined hostname, and if possible, send the received response body back to the caller.

The purpose of this script is to allow sending http requests to other ports than 80 for users that have a firewall or proxy server that only allow http request to port 80.

What is the easiest way to write such a script?
0
Comment
Question by:rj2
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 18
  • 14
32 Comments
 
LVL 18

Expert Comment

by:kandura
ID: 12170039
Where is your script going to be running? It sounds like you want it on the client machines, and then you'd have to make them point their browser to use your script as a proxy, which in turn uses their "real" proxy as proxy.

Let's see if I can make a drawing of sorts.

+----------+
| browser  |
+---+------+
    |
    v
+---+------+
| script   |   <--- is this your intended position?
+---+------+
    |
    v
+---+------+
| proxy    |
+---+------+
    |
    v
+---+------+
| internet |
+---+------+


In any case it sounds like your best bet is still a script based on HTTP::Proxy.
0
 
LVL 10

Author Comment

by:rj2
ID: 12170117
No, after the proxy. It is actually the proxy that denies requests to ports other than 80. And the proxy settings can not be changed in certain environments.
+----------+
| browser  |
+---+------+
    |
    v
+---+------+
| proxy    |
+---+------+
    |
    v
+---+------+
| script   |  
+---+------+
    |
    v
+---+------+
| internet |
+---+------+
0
 
LVL 18

Expert Comment

by:kandura
ID: 12170190
I believe that that implies you will have to reconfigure the proxy to use your script as its proxy. And since your script is merely redirecting requests, it's natural to think of it as a proxy.
Implementing it with HTTP::Proxy should be easy enough. What exactly is that host doing to which you want to redirect all requests?
0
What does it mean to be "Always On"?

Is your cloud always on? With an Always On cloud you won't have to worry about downtime for maintenance or software application code updates, ensuring that your bottom line isn't affected.

 
LVL 10

Author Comment

by:rj2
ID: 12170540
A client want to upload data a server (server 1). This server accepts data on a port other than 80. The client uses a proxy, and the proxy can not be changed. But I can change where the data is uploaded to, so it is uploaded to server 2 instead, where I place my script. So to enable the client to upload data to server1 I was wondering if I could write a script and place it on server 2 that resends the http requests to server 1.
0
 
LVL 18

Expert Comment

by:kandura
ID: 12170676
So you would map requests for server1 to server2 on the proxy, right?
Can't you simply map those requests to the correct port then?

Just so that we're clear on this: you do have control over the proxy machine, but not over the clients behind that proxy. How much control do you have? Could you map requests for server1 to (say) server1:8080 ?
0
 
LVL 10

Author Comment

by:rj2
ID: 12171060
I have control over the client software running at the user's computer. This client software needs to send request to a different port than 80, but the proxy will not do this.
I have no control over the proxy.
So I thought maybe I could setup a script on port 80 on a different server, reconfigurate the client software to talk to this script and let the script resend the data to the final destination (on different port than 80).
0
 
LVL 18

Expert Comment

by:kandura
ID: 12172950
Yes, now it makes sense. That should be perfectly possible, I think.
You configure the client app to communicate with server2 over port 80, which is your yet_to_be_created perl script, which proxies all those requests to the intended server1 on the port you need.
Although the "proxy" functionality of your script is very limited, I'd still suggest using HTTP::Proxy, since it seems to wrap up most of the requirements you have.
0
 
LVL 10

Author Comment

by:rj2
ID: 12177110
ok, do you got some sample code to get me started?
0
 
LVL 10

Author Comment

by:rj2
ID: 12186871
How do I tell Http::Proxy that all requests should be forwarded to a specific, predefined host?
Or is there other modules better suited for this task?
0
 
LVL 18

Expert Comment

by:kandura
ID: 12187910
I'm sorry, I haven't had time to look into it in detail. I had wanted to create a test script, so I tried to install HTTP::Proxy, but that failed on my machine, and I had a lot of other work to do.
I believe you have to add a HTTP::Proxy::HeaderFilter that rewrites the host part of the request, which, looking at the documentation, should be straightforward.
0
 
LVL 10

Author Comment

by:rj2
ID: 12206401
Please post some sample code when you get around to it then.
0
 
LVL 10

Author Comment

by:rj2
ID: 12224586
Sample code? Anyone?
0
 
LVL 18

Expert Comment

by:kandura
ID: 12231621
rj2, sorry I've been so unresponsive, but I had two projects go live at the same time...

I tried the following script, but it didn't seem to work. I only got 501 errors :(

Couldn't you simply use url rewriting on server1? Do you have a webserver running on that server?

Another thing you might look into is Net::HTTPServer.

#!/usr/bin/perl

use strict;

use HTTP::Proxy qw/ALL/;
use HTTP::Proxy::HeaderFilter::simple;

# initialisation
my $proxy = HTTP::Proxy->new( port => 80 );
$proxy->host(undef);
$proxy->push_filter( request => HTTP::Proxy::HeaderFilter::simple->new(
        sub {
                $_[1]->header( Host => 'server2:port2' );
        }));
$proxy->logmask( ALL );

# this is a MainLoop-like method
$proxy->start;
0
 
LVL 10

Author Comment

by:rj2
ID: 12236491
Server1 is a Windows server with IIS, not Apache unfortunately.
Is URL rewriting supported on IIS?
0
 
LVL 18

Expert Comment

by:kandura
ID: 12239158
Not really. You can redirect some url's, and pass on some parameters, but it's not nearly as full featured. And I doubt it will do POST requests.

I found someone blogging about a mod_rewrite for IIS: http://kalsey.com/2002/03/mod_rewrite_for_iis/
It may well be worth it to see about the free version.
Here's a list of other available tools: http://www.jdhodges.com/log/1309

0
 
LVL 10

Author Comment

by:rj2
ID: 12249407
Ok, I must check further, they might have objections to installing such extenions on the production server, must probably tested first.

Is it possible to use just the CGI and LWP modules to resend http requests like this?
You do know both the request method and which parameters you received, so this can be used to build a new http request?
0
 
LVL 18

Expert Comment

by:kandura
ID: 12258253
rj2,
> Is it possible to use just the CGI and LWP modules to resend http requests
> like this?
> You do know both the request method and which parameters you received,
> so this can be used to build a new http request?

Of course, but that is such a lot of work ;) I had hoped to find you an answer that works out of the box,
but it seems there's nothing left but do it ourselves.

Some of the biggest obstacles I'm foreseeing are:
- request header rewriting, both request uri and host header
- response rewriting of url's
- inefficiencies in file upload and general POST requests:
    we need to retrieve the entire request before we can retransmit

Of course, these can be overcome, but that probably involves quite a bit of work.

The general process would look something like the following script. I have tested it on a few GET requests, but you should probably verify that the headers are correct in other requests.

#!/usr/bin/perl

use HTTP::Request;
use LWP::UserAgent;
use URI;

my $remote_host = 'server2';
my $remote_port = 8080;
my $remote_uri  = "http::/$remote_host:$remote_port/cgi-bin/your_script.pl";

# construct new HTTP::Request from headers
my $req         = get_request();

# create a user agent
my $ua = LWP::UserAgent->new( # attrs
        keep_alive => 2,
        timeout    => 120,
    );

# get the request
my $resp = $ua->request($req);
print $resp->as_string;

sub get_request
{
    my $uri = URI->new($remote_uri);
    $uri->query($ENV{'QUERY_STRING'});

    my $req = new HTTP::Request($ENV{'REQUEST_METHOD'}, $uri);
    $req->header(
        Accept          => $ENV{'HTTP_ACCEPT'},
        Accept_Charset  => $ENV{'HTTP_ACCEPT_CHARSET'},
        Accept_Encoding => $ENV{'HTTP_ACCEPT_ENCODING'},
        Accept_Language => $ENV{'HTTP_ACCEPT_LANGUAGE'},
        Cookie          => $ENV{'HTTP_COOKIE'},
        Host            => $remote_host,
        User_Agent      => $ENV{'HTTP_USER_AGENT'},
    );

    $req->content(join('', <STDIN>)); # for POSTs. Please test!
    return $req;
}
0
 
LVL 18

Expert Comment

by:kandura
ID: 12342751
rj2, any news on this?
0
 
LVL 10

Author Comment

by:rj2
ID: 12344918
The Apache log says "malformed header from script. Bad header=HTTP/1.1 401 (Unauthorized) " when I tested it
0
 
LVL 10

Author Comment

by:rj2
ID: 12344928
Browser says "http 500 internal server error" when I tested it.
0
 
LVL 18

Expert Comment

by:kandura
ID: 12345727
That's not a whole lot of information to go on, but it sounds like your remote script requires authentication.
You probably have to add the headers for that in the get_request sub.
And you may need to either run this script in nph mode, or strip the first header line (with "HTTP/1.x") from the response.
0
 
LVL 10

Author Comment

by:rj2
ID: 12347630
Hello again!
I tried to add Authorization header and rename script to nph-script.pl and that helped.
Just one odd thing, I tested with a GET request against a script that displays a form
The final end html tag is not displayed when I call this redirection script.
The last html that is printed by the redirection script is
</form></body><
it should hav been
</form></body></html>
So the last 6 characters are not displayed. The final </html> tag is displayed correctly when I point browser directly to the final destination.
0
 
LVL 18

Expert Comment

by:kandura
ID: 12351932
That's weird. It works fine on my test scripts. Can you dump both the request and the response to a file, and see what's in there?
0
 
LVL 10

Author Comment

by:rj2
ID: 12355207
Yes, it is weird. I noticed that if the final destination script output less data, the problem does not occur. So the problem only occurs when the final destination script outputs a certain amount of data.
First, this is a dump of the request when I point the webbrowser directly to the final destination. No problem here, the final </html> end tag is displayed as expected.

07:42:03.531  !
07:42:03.531  !
07:42:03.531  GET /cgi-test/testupload.pl HTTP/1.1
              Accept: */*
              Accept-Language: no
              Accept-Encoding: gzip, deflate
              User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1
              .4322)
              Host: 10.0.1.5
              Connection: Keep-Alive
07:42:03.531  !
07:42:03.531  !
07:42:03.734  HTTP/1.1 200 OK
              Date: Wed, 20 Oct 2004 05:35:52 GMT
              Server: Apache/2.0.50 (Win32) PHP/4.3.8
              Content-Length: 797
              Keep-Alive: timeout=15, max=100
              Connection: Keep-Alive
              Content-Type: text/html; charset=ISO-8859-1
             
              <?xml version="1.0" encoding="iso-8859-1"?>
              <!DOCTYPE html
              .PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
              . "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
              <html xmlns="http://www.w3.org/1999/xhtml" lang="en-US" xml:lang="en-US"><head><
              title>File upload test</title>
              </head><body><form method="post" action="/cgi-test/testupload.pl" enctype="multi
              part/form-data">
              Operation:<input type="text" name="operation"  size="60" /><br />Localfile:<inpu
              t type="file" name="localfile"  size="60" /><br />Remotefile:<input type="text"
              name="remotefile"  size="60" /><br />Remotedir:<input type="text" name="remotedi
              r"  size="60" /><br />Remotenewfile:<input type="text" name="remotenewfile"  siz
              e="60" /><br /><input type="submit" name=".submit" value="Upload" /><div></div><
              /form></body></html>



Here is a dump of what happens if I point browser to the redirection script.

07:44:46.546  !
07:44:46.546  !
07:44:46.546  GET /cgi-bin/nph-resend.pl HTTP/1.1
              Accept: */*
              Accept-Language: no
              Accept-Encoding: gzip, deflate
              User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1
              .4322)
              Host: localhost
              Connection: Keep-Alive
07:44:46.546  !
07:44:46.562  !
07:44:47.046  HTTP/1.1 200 OK
              Connection: Keep-Alive
              Date: Wed, 20 Oct 2004 05:38:35 GMT
              Server: Apache/2.0.50 (Win32) PHP/4.3.8
              Content-Length: 798
              Content-Type: text/html; charset=ISO-8859-1
              Client-Date: Wed, 20 Oct 2004 05:44:47 GMT
              Client-Peer: 10.0.1.5:80
              Client-Response-Num: 1
              Keep-Alive: timeout=15, max=100
              Title: File upload test
             
              <?xml version="1.0" encoding="iso-8859-1"?>
              <!DOCTYPE html
              .PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
              . "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
              <html xmlns="http://www.w3.org/1999/xhtml" lang="en-US" xml:lang="en-US"><head><
              title>File upload test</title>
              </head><body><form method="post" action="/cgi-test/testupload.pl?" enctype="mult
              ipart/form-data">
              Operation:<input type="text" name="operation"  size="60" /><br />Localfile:<inpu
              t type="file" name="localfile"  size="60" /><br />Remotefile:<input type="text"
              name="remotefile"  size="60" /><br />Remotedir:<input type="text" name="remotedi
              r"  size="60" /><br />Remotenewfile:<input type="text" name="
07:44:47.062  remotenewfile"  size="60" /><br /><input type="submit" name=".submit" value="Upl
              oad" /><div></div></form></body><
07:44:47.062  /
07:44:52.031  /
07:44:57.031  /

0
 
LVL 10

Author Comment

by:rj2
ID: 12368241
When the problem occurs, if I press refresh in the browser the missing text "/html>" is then displayed as the only text on the page.
I tried to add $| = 1; as the first thing in the script, but it did not seem to help much.
0
 
LVL 10

Author Comment

by:rj2
ID: 12368276
If I change the final destination script so that it output a lot more data (like 300kb), the exact same text is still missing.
So it is the last few bytes that are missing also if there is alot more data.
0
 
LVL 10

Author Comment

by:rj2
ID: 12420162
kandura, any news on this?
0
 
LVL 18

Expert Comment

by:kandura
ID: 12435189
Hi rj2,

Again sorry for the long delay. We're going to move to another country in 6 weeks, I got married in the mean time, and we have had a lot to organize lately.

Can you have the proxy script dump its $response as well? I wonder whether it doesn't *receive* the complete response, or whether it doesn't *send* the complete response.
I haven't been able to reproduce this issue, my tests all went perfectly. In fact, I use a similar script in one of our web apps to serve files to subscribers. The external webserver retrieves the files from an internal file server through HTTP (using LWP etc.), and I haven't had a single problem yet. Since this concerns mainly images, any missing data would be obvious immediately.
0
 
LVL 10

Author Comment

by:rj2
ID: 12441792
Hello Kandura,

That is some really life-changing events, no wonder you're busy.

I added code like "open(DEBUG,">debug.txt");print DEBUG $resp->as_string;close(DEBUG);" to dump the response to file.
The last output to the file is "</form></body></html>", so the end html tag is being printed.
But when I select "view source" in the browser, the last data is "</form></body><"
0
 
LVL 18

Accepted Solution

by:
kandura earned 2000 total points
ID: 12473026
So either the script doesn't pass on all the data, or the browser doesn't receive all of it.
The next step I think would be to check whether the Content-Length header still matches the truth. You might try to change that header to the length of $resp->content.
Another idea is to make sure all output is flushed, by setting $|=1 somewhere near the top of the script, but I seem to recall you already tried that. It might help to add a couple of newlines to the output, or maybe even close STDOUT at the end.

0
 
LVL 10

Author Comment

by:rj2
ID: 12482009
Setting the Content-Length header fixed it.

Thanks for all the help and good luck moving, probably very interesting to try living in a different country for some time.
0
 
LVL 18

Expert Comment

by:kandura
ID: 12482050
Excellent! Again, sorry it took little over a month to get this sorted, but glad I could be of help.

Yes, I'm pretty excited about moving. "May you live in interesting times" ;^)
0

Featured Post

Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Many time we need to work with multiple files all together. If its windows system then we can use some GUI based editor to accomplish our task. But what if you are on putty or have only CLI(Command Line Interface) as an option to  edit your files. I…
There are many situations when we need to display the data in sorted order. For example: Student details by name or by rank or by total marks etc. If you are working on data driven based projects then you will use sorting techniques very frequently.…
Explain concepts important to validation of email addresses with regular expressions. Applies to most languages/tools that uses regular expressions. Consider email address RFCs: Look at HTML5 form input element (with type=email) regex pattern: T…
Six Sigma Control Plans

610 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question