Solved

Http resend script question

Posted on 2004-09-27
32
304 Views
Last Modified: 2008-02-01
Hello!
I would like to write a script that resends all http requests to a single, predefined hostname with the exact same same parameters that the script receives. The script must handle HEAD, GET and POST requests. For file uploads, the script could receive the incoming file, resend the incoming file to the predefined hostname, and if possible, send the received response body back to the caller.

The purpose of this script is to allow sending http requests to other ports than 80 for users that have a firewall or proxy server that only allow http request to port 80.

What is the easiest way to write such a script?
0
Comment
Question by:rj2
  • 18
  • 14
32 Comments
 
LVL 18

Expert Comment

by:kandura
Comment Utility
Where is your script going to be running? It sounds like you want it on the client machines, and then you'd have to make them point their browser to use your script as a proxy, which in turn uses their "real" proxy as proxy.

Let's see if I can make a drawing of sorts.

+----------+
| browser  |
+---+------+
    |
    v
+---+------+
| script   |   <--- is this your intended position?
+---+------+
    |
    v
+---+------+
| proxy    |
+---+------+
    |
    v
+---+------+
| internet |
+---+------+


In any case it sounds like your best bet is still a script based on HTTP::Proxy.
0
 
LVL 10

Author Comment

by:rj2
Comment Utility
No, after the proxy. It is actually the proxy that denies requests to ports other than 80. And the proxy settings can not be changed in certain environments.
+----------+
| browser  |
+---+------+
    |
    v
+---+------+
| proxy    |
+---+------+
    |
    v
+---+------+
| script   |  
+---+------+
    |
    v
+---+------+
| internet |
+---+------+
0
 
LVL 18

Expert Comment

by:kandura
Comment Utility
I believe that that implies you will have to reconfigure the proxy to use your script as its proxy. And since your script is merely redirecting requests, it's natural to think of it as a proxy.
Implementing it with HTTP::Proxy should be easy enough. What exactly is that host doing to which you want to redirect all requests?
0
 
LVL 10

Author Comment

by:rj2
Comment Utility
A client want to upload data a server (server 1). This server accepts data on a port other than 80. The client uses a proxy, and the proxy can not be changed. But I can change where the data is uploaded to, so it is uploaded to server 2 instead, where I place my script. So to enable the client to upload data to server1 I was wondering if I could write a script and place it on server 2 that resends the http requests to server 1.
0
 
LVL 18

Expert Comment

by:kandura
Comment Utility
So you would map requests for server1 to server2 on the proxy, right?
Can't you simply map those requests to the correct port then?

Just so that we're clear on this: you do have control over the proxy machine, but not over the clients behind that proxy. How much control do you have? Could you map requests for server1 to (say) server1:8080 ?
0
 
LVL 10

Author Comment

by:rj2
Comment Utility
I have control over the client software running at the user's computer. This client software needs to send request to a different port than 80, but the proxy will not do this.
I have no control over the proxy.
So I thought maybe I could setup a script on port 80 on a different server, reconfigurate the client software to talk to this script and let the script resend the data to the final destination (on different port than 80).
0
 
LVL 18

Expert Comment

by:kandura
Comment Utility
Yes, now it makes sense. That should be perfectly possible, I think.
You configure the client app to communicate with server2 over port 80, which is your yet_to_be_created perl script, which proxies all those requests to the intended server1 on the port you need.
Although the "proxy" functionality of your script is very limited, I'd still suggest using HTTP::Proxy, since it seems to wrap up most of the requirements you have.
0
 
LVL 10

Author Comment

by:rj2
Comment Utility
ok, do you got some sample code to get me started?
0
 
LVL 10

Author Comment

by:rj2
Comment Utility
How do I tell Http::Proxy that all requests should be forwarded to a specific, predefined host?
Or is there other modules better suited for this task?
0
 
LVL 18

Expert Comment

by:kandura
Comment Utility
I'm sorry, I haven't had time to look into it in detail. I had wanted to create a test script, so I tried to install HTTP::Proxy, but that failed on my machine, and I had a lot of other work to do.
I believe you have to add a HTTP::Proxy::HeaderFilter that rewrites the host part of the request, which, looking at the documentation, should be straightforward.
0
 
LVL 10

Author Comment

by:rj2
Comment Utility
Please post some sample code when you get around to it then.
0
 
LVL 10

Author Comment

by:rj2
Comment Utility
Sample code? Anyone?
0
 
LVL 18

Expert Comment

by:kandura
Comment Utility
rj2, sorry I've been so unresponsive, but I had two projects go live at the same time...

I tried the following script, but it didn't seem to work. I only got 501 errors :(

Couldn't you simply use url rewriting on server1? Do you have a webserver running on that server?

Another thing you might look into is Net::HTTPServer.

#!/usr/bin/perl

use strict;

use HTTP::Proxy qw/ALL/;
use HTTP::Proxy::HeaderFilter::simple;

# initialisation
my $proxy = HTTP::Proxy->new( port => 80 );
$proxy->host(undef);
$proxy->push_filter( request => HTTP::Proxy::HeaderFilter::simple->new(
        sub {
                $_[1]->header( Host => 'server2:port2' );
        }));
$proxy->logmask( ALL );

# this is a MainLoop-like method
$proxy->start;
0
 
LVL 10

Author Comment

by:rj2
Comment Utility
Server1 is a Windows server with IIS, not Apache unfortunately.
Is URL rewriting supported on IIS?
0
 
LVL 18

Expert Comment

by:kandura
Comment Utility
Not really. You can redirect some url's, and pass on some parameters, but it's not nearly as full featured. And I doubt it will do POST requests.

I found someone blogging about a mod_rewrite for IIS: http://kalsey.com/2002/03/mod_rewrite_for_iis/
It may well be worth it to see about the free version.
Here's a list of other available tools: http://www.jdhodges.com/log/1309

0
 
LVL 10

Author Comment

by:rj2
Comment Utility
Ok, I must check further, they might have objections to installing such extenions on the production server, must probably tested first.

Is it possible to use just the CGI and LWP modules to resend http requests like this?
You do know both the request method and which parameters you received, so this can be used to build a new http request?
0
Free Trending Threat Insights Every Day

Enhance your security with threat intelligence from the web. Get trending threat insights on hackers, exploits, and suspicious IP addresses delivered to your inbox with our free Cyber Daily.

 
LVL 18

Expert Comment

by:kandura
Comment Utility
rj2,
> Is it possible to use just the CGI and LWP modules to resend http requests
> like this?
> You do know both the request method and which parameters you received,
> so this can be used to build a new http request?

Of course, but that is such a lot of work ;) I had hoped to find you an answer that works out of the box,
but it seems there's nothing left but do it ourselves.

Some of the biggest obstacles I'm foreseeing are:
- request header rewriting, both request uri and host header
- response rewriting of url's
- inefficiencies in file upload and general POST requests:
    we need to retrieve the entire request before we can retransmit

Of course, these can be overcome, but that probably involves quite a bit of work.

The general process would look something like the following script. I have tested it on a few GET requests, but you should probably verify that the headers are correct in other requests.

#!/usr/bin/perl

use HTTP::Request;
use LWP::UserAgent;
use URI;

my $remote_host = 'server2';
my $remote_port = 8080;
my $remote_uri  = "http::/$remote_host:$remote_port/cgi-bin/your_script.pl";

# construct new HTTP::Request from headers
my $req         = get_request();

# create a user agent
my $ua = LWP::UserAgent->new( # attrs
        keep_alive => 2,
        timeout    => 120,
    );

# get the request
my $resp = $ua->request($req);
print $resp->as_string;

sub get_request
{
    my $uri = URI->new($remote_uri);
    $uri->query($ENV{'QUERY_STRING'});

    my $req = new HTTP::Request($ENV{'REQUEST_METHOD'}, $uri);
    $req->header(
        Accept          => $ENV{'HTTP_ACCEPT'},
        Accept_Charset  => $ENV{'HTTP_ACCEPT_CHARSET'},
        Accept_Encoding => $ENV{'HTTP_ACCEPT_ENCODING'},
        Accept_Language => $ENV{'HTTP_ACCEPT_LANGUAGE'},
        Cookie          => $ENV{'HTTP_COOKIE'},
        Host            => $remote_host,
        User_Agent      => $ENV{'HTTP_USER_AGENT'},
    );

    $req->content(join('', <STDIN>)); # for POSTs. Please test!
    return $req;
}
0
 
LVL 18

Expert Comment

by:kandura
Comment Utility
rj2, any news on this?
0
 
LVL 10

Author Comment

by:rj2
Comment Utility
The Apache log says "malformed header from script. Bad header=HTTP/1.1 401 (Unauthorized) " when I tested it
0
 
LVL 10

Author Comment

by:rj2
Comment Utility
Browser says "http 500 internal server error" when I tested it.
0
 
LVL 18

Expert Comment

by:kandura
Comment Utility
That's not a whole lot of information to go on, but it sounds like your remote script requires authentication.
You probably have to add the headers for that in the get_request sub.
And you may need to either run this script in nph mode, or strip the first header line (with "HTTP/1.x") from the response.
0
 
LVL 10

Author Comment

by:rj2
Comment Utility
Hello again!
I tried to add Authorization header and rename script to nph-script.pl and that helped.
Just one odd thing, I tested with a GET request against a script that displays a form
The final end html tag is not displayed when I call this redirection script.
The last html that is printed by the redirection script is
</form></body><
it should hav been
</form></body></html>
So the last 6 characters are not displayed. The final </html> tag is displayed correctly when I point browser directly to the final destination.
0
 
LVL 18

Expert Comment

by:kandura
Comment Utility
That's weird. It works fine on my test scripts. Can you dump both the request and the response to a file, and see what's in there?
0
 
LVL 10

Author Comment

by:rj2
Comment Utility
Yes, it is weird. I noticed that if the final destination script output less data, the problem does not occur. So the problem only occurs when the final destination script outputs a certain amount of data.
First, this is a dump of the request when I point the webbrowser directly to the final destination. No problem here, the final </html> end tag is displayed as expected.

07:42:03.531  !
07:42:03.531  !
07:42:03.531  GET /cgi-test/testupload.pl HTTP/1.1
              Accept: */*
              Accept-Language: no
              Accept-Encoding: gzip, deflate
              User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1
              .4322)
              Host: 10.0.1.5
              Connection: Keep-Alive
07:42:03.531  !
07:42:03.531  !
07:42:03.734  HTTP/1.1 200 OK
              Date: Wed, 20 Oct 2004 05:35:52 GMT
              Server: Apache/2.0.50 (Win32) PHP/4.3.8
              Content-Length: 797
              Keep-Alive: timeout=15, max=100
              Connection: Keep-Alive
              Content-Type: text/html; charset=ISO-8859-1
             
              <?xml version="1.0" encoding="iso-8859-1"?>
              <!DOCTYPE html
              .PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
              . "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
              <html xmlns="http://www.w3.org/1999/xhtml" lang="en-US" xml:lang="en-US"><head><
              title>File upload test</title>
              </head><body><form method="post" action="/cgi-test/testupload.pl" enctype="multi
              part/form-data">
              Operation:<input type="text" name="operation"  size="60" /><br />Localfile:<inpu
              t type="file" name="localfile"  size="60" /><br />Remotefile:<input type="text"
              name="remotefile"  size="60" /><br />Remotedir:<input type="text" name="remotedi
              r"  size="60" /><br />Remotenewfile:<input type="text" name="remotenewfile"  siz
              e="60" /><br /><input type="submit" name=".submit" value="Upload" /><div></div><
              /form></body></html>



Here is a dump of what happens if I point browser to the redirection script.

07:44:46.546  !
07:44:46.546  !
07:44:46.546  GET /cgi-bin/nph-resend.pl HTTP/1.1
              Accept: */*
              Accept-Language: no
              Accept-Encoding: gzip, deflate
              User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1
              .4322)
              Host: localhost
              Connection: Keep-Alive
07:44:46.546  !
07:44:46.562  !
07:44:47.046  HTTP/1.1 200 OK
              Connection: Keep-Alive
              Date: Wed, 20 Oct 2004 05:38:35 GMT
              Server: Apache/2.0.50 (Win32) PHP/4.3.8
              Content-Length: 798
              Content-Type: text/html; charset=ISO-8859-1
              Client-Date: Wed, 20 Oct 2004 05:44:47 GMT
              Client-Peer: 10.0.1.5:80
              Client-Response-Num: 1
              Keep-Alive: timeout=15, max=100
              Title: File upload test
             
              <?xml version="1.0" encoding="iso-8859-1"?>
              <!DOCTYPE html
              .PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
              . "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
              <html xmlns="http://www.w3.org/1999/xhtml" lang="en-US" xml:lang="en-US"><head><
              title>File upload test</title>
              </head><body><form method="post" action="/cgi-test/testupload.pl?" enctype="mult
              ipart/form-data">
              Operation:<input type="text" name="operation"  size="60" /><br />Localfile:<inpu
              t type="file" name="localfile"  size="60" /><br />Remotefile:<input type="text"
              name="remotefile"  size="60" /><br />Remotedir:<input type="text" name="remotedi
              r"  size="60" /><br />Remotenewfile:<input type="text" name="
07:44:47.062  remotenewfile"  size="60" /><br /><input type="submit" name=".submit" value="Upl
              oad" /><div></div></form></body><
07:44:47.062  /
07:44:52.031  /
07:44:57.031  /

0
 
LVL 10

Author Comment

by:rj2
Comment Utility
When the problem occurs, if I press refresh in the browser the missing text "/html>" is then displayed as the only text on the page.
I tried to add $| = 1; as the first thing in the script, but it did not seem to help much.
0
 
LVL 10

Author Comment

by:rj2
Comment Utility
If I change the final destination script so that it output a lot more data (like 300kb), the exact same text is still missing.
So it is the last few bytes that are missing also if there is alot more data.
0
 
LVL 10

Author Comment

by:rj2
Comment Utility
kandura, any news on this?
0
 
LVL 18

Expert Comment

by:kandura
Comment Utility
Hi rj2,

Again sorry for the long delay. We're going to move to another country in 6 weeks, I got married in the mean time, and we have had a lot to organize lately.

Can you have the proxy script dump its $response as well? I wonder whether it doesn't *receive* the complete response, or whether it doesn't *send* the complete response.
I haven't been able to reproduce this issue, my tests all went perfectly. In fact, I use a similar script in one of our web apps to serve files to subscribers. The external webserver retrieves the files from an internal file server through HTTP (using LWP etc.), and I haven't had a single problem yet. Since this concerns mainly images, any missing data would be obvious immediately.
0
 
LVL 10

Author Comment

by:rj2
Comment Utility
Hello Kandura,

That is some really life-changing events, no wonder you're busy.

I added code like "open(DEBUG,">debug.txt");print DEBUG $resp->as_string;close(DEBUG);" to dump the response to file.
The last output to the file is "</form></body></html>", so the end html tag is being printed.
But when I select "view source" in the browser, the last data is "</form></body><"
0
 
LVL 18

Accepted Solution

by:
kandura earned 500 total points
Comment Utility
So either the script doesn't pass on all the data, or the browser doesn't receive all of it.
The next step I think would be to check whether the Content-Length header still matches the truth. You might try to change that header to the length of $resp->content.
Another idea is to make sure all output is flushed, by setting $|=1 somewhere near the top of the script, but I seem to recall you already tried that. It might help to add a couple of newlines to the output, or maybe even close STDOUT at the end.

0
 
LVL 10

Author Comment

by:rj2
Comment Utility
Setting the Content-Length header fixed it.

Thanks for all the help and good luck moving, probably very interesting to try living in a different country for some time.
0
 
LVL 18

Expert Comment

by:kandura
Comment Utility
Excellent! Again, sorry it took little over a month to get this sorted, but glad I could be of help.

Yes, I'm pretty excited about moving. "May you live in interesting times" ;^)
0

Featured Post

Highfive Gives IT Their Time Back

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

Join & Write a Comment

I've just discovered very important differences between Windows an Unix formats in Perl,at least 5.xx.. MOST IMPORTANT: Use Unix file format while saving Your script. otherwise it will have ^M s or smth likely weird in the EOL, Then DO NOT use m…
Checking the Alert Log in AWS RDS Oracle can be a pain through their user interface.  I made a script to download the Alert Log, look for errors, and email me the trace files.  In this article I'll describe what I did and share my script.
Explain concepts important to validation of email addresses with regular expressions. Applies to most languages/tools that uses regular expressions. Consider email address RFCs: Look at HTML5 form input element (with type=email) regex pattern: T…
In this seventh video of the Xpdf series, we discuss and demonstrate the PDFfonts utility, which lists all the fonts used in a PDF file. It does this via a command line interface, making it suitable for use in programs, scripts, batch files — any pl…

744 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

15 Experts available now in Live!

Get 1:1 Help Now