Solved

PHP site mirror

Posted on 2003-10-22
30
565 Views
Last Modified: 2011-10-03
I am looking for a script to create a mirror of a site, so it simply checks the URL then downloads a file from anouther server and serves that to the user, in effect i want to access anouther server from my URL... so if i type:

http://www.myserver.com/some/page.htm

it will load

http://www.someotherserver.com/some/page.htm

the main reason i want it is because i am running 2 web servers but only have 1 IP so i want the web server that has port 80 forwarded to it to simply fetch from the other server, (completely invisible to the user) at the moment i run it off of port 8080, but it would be so much better if i could run if from port 80, i am of course using host headers on the web server with port 80 open to detect if its a site for it to serve or if it should go to the other server.
0
Comment
Question by:123webserver
  • 9
  • 8
  • 6
  • +1
30 Comments
 

Author Comment

by:123webserver
ID: 9602915
oh, i have *.mydomain.com set to point to my IP, and the reason i want this is because my mail server has web access and i want to use mail.mydomain.com to access the mail servers web login, i thought about simply creating (or downloading) anouther web script for connecting to the mail server via pop3 or something but decided the one that it comes with it more reliable.
0
 
LVL 8

Expert Comment

by:inq123
ID: 9603218
Hi 123webserver,

This is a perfect job for mod_rewrite for apache.  My strategy is to use mod_rewrite rule trap ANY URL and send the path info to a php script, which then either forward the request, or open that URL and streams results back to user.  Isn't that a smart strategy! :-)  It definitely works very well for your situation.

Cheers!
0
 
LVL 1

Accepted Solution

by:
fyuber earned 65 total points
ID: 9603546
If you want do it manually, you can use $_SERVER["PHP_SELF"] and file_get_contents()
0
 

Author Comment

by:123webserver
ID: 9605396
how would i use...

$_SERVER["PHP_SELF"]
and
file_get_contents()

and in reference to the apache mod, i'm not using apache :-)
0
 
LVL 6

Expert Comment

by:aolXFT
ID: 9605685
Just put the following into your PHP Script:

<?php
echo file_get_contents("http://www.someotherserver.com" . $_SERVER['PHP_SELF']);
?>

If PHP_SELF does not fit your needs check the php_info page for a better variable. I'd personally recommend REQUEST_URI. This however may be specific to CGI, and Apache(which is what I'm using - blame the admin for the CGI not me :|), so you should check it out.

I recommend REQUEST_URI because it also passes GET Args.
0
 

Author Comment

by:123webserver
ID: 9605715
I managed to get it to fetch google.com with that, but how would i make it so that it gets the final part of the URL from the URL entered (i am setting the php script as the 404 error message) and to then add the URL to the end, eg if i type www.myserver.com/something.htm  it will pick up the /something.htm and add it to the end of www.otherserver.com

And i would also need to replace any occourances of www.otherserver.com with myserver.com although i believe most of the links are relative anyway so wouldn't matter
0
 

Author Comment

by:123webserver
ID: 9605717
I managed to get it to fetch google.com with that, but how would i make it so that it gets the final part of the URL from the URL entered (i am setting the php script as the 404 error message) and to then add the URL to the end, eg if i type www.myserver.com/something.htm  it will pick up the /something.htm and add it to the end of www.otherserver.com

And i would also need to replace any occourances of www.otherserver.com with myserver.com although i believe most of the links are relative anyway so wouldn't matter
0
 
LVL 6

Expert Comment

by:aolXFT
ID: 9605742
use $_SERVER['SCRIPT_URI']

SCRIPT_URI, Does not include the domain name, and only the end, of the url.

<?php

$url = "http://www.otherserver.com" . $_SERVER['SCRIPT_URI'];

?>

0
 
LVL 8

Expert Comment

by:inq123
ID: 9605811
> And i would also need to replace any occourances of www.otherserver.com with myserver.com although i believe most of the links are relative anyway so wouldn't matter

I told you to use mod_rewrite to do that.  I'll post a complete set of code when I have time.
0
 
LVL 8

Expert Comment

by:inq123
ID: 9606740
123webserver,

My suggestion worked out even better than I thought.  Here's the fully tested code for you.  First save the following code as .htaccess in your server's root document directory (htdocs on apache):

Options +FollowSymLinks
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteRule ^(.*)$      redirect.php?url=$1  [QSA]
</IfModule>

Now save this code as redirect.php in the root dir (same as the .htaccess):

<?
  header("Location: http://www.yahoo.com/" . $_REQUEST['url']);
?>

Now type any request to your server like http://www.myserver.com/r/xn, and you'll be redirected to http://www.yahoo.com/r/xn which is http://news.yahoo.com.  Your site essentially became yahoo's mirror!  But there's more, if you do not want user even realize that you're mirroring something, change the redirect.php code to this:

<?
  $lines = file("http://www.yahoo.com/" . $_REQUEST['url']);
  foreach ($lines as $line)
  {
    echo $line;
  }
?>

Now your site is a true mirror that nobody even realize it's a mirror.  For your problem just change the www.myserver.com to your server, and set port correctly in the redirect.php.

Using my method, the entire yahoo site is mirrored using only 2 files with less than 10 lines of code.  Isn't that a good way of mirroring? (sorry can't help gloating a bit 'cause I like this solution a lot)

inq
0
 
LVL 8

Expert Comment

by:inq123
ID: 9606772
BTW, two things:

1. Your apache server must have mod_rewrite to have it work.  But once you have this module, it works so well!
2. The reason I used file instead of file_get_contents is that the latter requires php >= 4.3.0 which is not the case for many shared hosts on the web.  If your server has such new php, then use this code in your redirect.php:

<?
  echo file_get_contents("http://www.yahoo.com/" . $_REQUEST['url']);
?>

which would make the code even shorter (and slightly faster I think).
0
 
LVL 8

Expert Comment

by:inq123
ID: 9606776
BTW, mod_rewrite is pretty standard in shared hosts, and it's very likely you have it.
0
 
LVL 6

Expert Comment

by:aolXFT
ID: 9607826
> 2. The reason I used file instead of file_get_contents is that the latter requires
 > php >= 4.3.0 which is not the case for many shared hosts on the web.  If your
 > server has such new php, then use this code in your redirect.php:

The reason I used file_get_contents, was because I wanted to give credit to fyuber, for mentioning it, and to avoid confusing the question more.

If you don't have file_get_contents, you should you should either write a file_get_contents, using fopen, and fread, or use fread, and fopen directly. Hosts that are not using 4.3.2 should be given a swift kick on the backside.

I think I wrote this for someone else here yesterday.

<?php
function file_get_contents($name){
       return fread( fopen($name, "w"), filesize($name) );
}
?>
0
Maximize Your Threat Intelligence Reporting

Reporting is one of the most important and least talked about aspects of a world-class threat intelligence program. Here’s how to do it right.

 

Author Comment

by:123webserver
ID: 9608053
thanks, i got the first one to work fine...

<?php
echo file_get_contents("http://www.google.com" . $_SERVER['REQUEST_URI']);
?>

seems to work, any idea on the replacing of the http://www.otherserver.com/  with http://www.myserver.com/ ?

and the idea on mod_rewrite wont work because i'm not using apache...
0
 

Author Comment

by:123webserver
ID: 9608123
I also noticed that information submitted to the script didn't work... DOH, (using POST, tried GET and it looked OK but not tested properly, havent gotten that far... lol)
0
 
LVL 8

Expert Comment

by:inq123
ID: 9608214
>the idea on mod_rewrite wont work because i'm not using apache...

Well, if you don't have it, would you please say so next time just so I don't waste the time working out a solution that won't work for you?  I would appreciate the courtesy.

As for aolXFT's comment: I wasn't referring to your post specifically at all, so no need to explain your choice to me or even OP.  But I agree with your suggestion of using fopen, that's probably faster.  But why do you use "w" flag instead of "r"?  A typo I guess.

Regardless, not using mod_rewrite (and apache) is really your loss, 'cause the suggestions above won't automatically mirror the whole site for you.  No way.
0
 
LVL 8

Expert Comment

by:inq123
ID: 9608260
BTW, even if you're using Windows, the Apache for windows works fine for Windows, 'cause I really see no other way how you can type in a random www.myserver.com/something.htm and have your php script picks that up and redirect it?  You simply have to have the web server process that URL differently and send the request to your php script, and that's what apache + mod_rewrite do.  Correct me if I'm wrong, but without using such a server-based mechanism, the mirroring effort simply won't succeed for an entire web site.
0
 

Author Comment

by:123webserver
ID: 9609170
I got it working fine, just need to sort out the replacing of http://www.server.com/  with http://www.myserver.com/
0
 

Author Comment

by:123webserver
ID: 9609174
and of course POST wont work... GET does though
0
 

Author Comment

by:123webserver
ID: 9610450
I got the following two to work exactly the way i wanted, and i was able to change the mail servers template so the hyperlink changing is no longer an issue, only thing i cant get is the damn POST data, when i use a form to POST data to the php script it doesn't get passed on to the other server, it just gets the page as if the data was never there.... GET works fine... if only i could change that in the templates... lol


<?php
echo file_get_contents("http://www.server.com" . $_SERVER['REQUEST_URI']);
?>


<?php
include ("http://www.server.com" . $_SERVER['REQUEST_URI']);  // found this one on google :-)
?>
0
 
LVL 1

Expert Comment

by:fyuber
ID: 9611195
Thanks aolxft for expanding my comment which I neglected to do. So you are using a 404 to retrieve the contents of the file requested from the other server.

I thought that your second server is invisible? If you want both domains to give the same results then maybe experiment with some virtual hosts.

If not apache what server are you using?

POST(/cookie/file upload) data I guess you can extract into the URL. Perhaps

foreach($_POST as $xxx){$url.=$xxx;}

inq, he did say he wasnt using apache before your suggestions (comment #4).

PS
Newest PHP is 4.3.4.
Myself, I would write a file_get_contents() using file().
0
 
LVL 8

Expert Comment

by:inq123
ID: 9611643
oops, you're right.  I have only myself to blame then. :-)  Still I can't see any way to do it better than I suggested.
0
 
LVL 6

Assisted Solution

by:aolXFT
aolXFT earned 65 total points
ID: 9616571
I haven't checked the PHP Source, but I'm led to believe that file_get_contents uses the method I outlined above, except in C, rather than PHP. Having that said, I'm not too sure.

If you want to get the post variables you have two options.

One of them is simple, and involves putting the post vars onto the QUERY_STRING, the other is a bit more complicated.

Way 1:
<?php echo file_get_contents("http://www.otherserver.com" . $_SERVER['REQUEST_URI'] . strpos($_SERVER['REQUEST_URI'], '?') ? '?':'&' . "postvars=" . serialize($_POST));

?>
If you can't understand this I'll explain better.


Then on the other Server, have at the start:

<?php

if($_SERVER['REMOTE_ADDR'] == "ins_ip_of_other_server_here"){
  $_POST = unserialize($_GET['postvars']);
}


As for the suggestion from google, about using include, I have one word: Don't.

It would be security suicide.

<?php
include ("http://www.server.com" . $_SERVER['REQUEST_URI']);  // found this one on google :-)
?>

I could call something like http://www.otherserver.com/@www.myhost.com/evil_php_instructions.txt

OK, I'm not exactly sure if that one would work, but you need to be very careful when including over http.

It did remind me though about readfile. Check out http://www.php.net/readfile

It is used similar to include, except that it does not evaluate the instructions it gets, and simply echos them straight out, which is the same as we've been doing with echo file_get_contents(), but is one command so it MAY(or may not) be faster, but I believe it would.
0
 
LVL 6

Expert Comment

by:aolXFT
ID: 9616587
Oh yeah, the other way involves making a socket connection to the server, and pass on all the headers.

I would recommend you either use PERL, or if there is a suitable Pear module, you could use that.

You should also check out cgi-proxy(from google). It's written in Perl, but designed to access sites indirectly. It may have what you need(and a lot more).  
0
 
LVL 1

Expert Comment

by:fyuber
ID: 9618422
Come to think about it actually readfile() would be better. And of course mod_rewrite even better.
I forgot where i read file_get_contents was based on file. It is of course "identical" except for the return value.
http://lxr.php.net/source/php-src/ext/standard/file.c line 436 for the source.
All this is getting really complicated for such a simple task. Maybe the web server could be port 80 and redirect mail requests to the other server. Or just use a different port for the mail web login.
0
 
LVL 6

Expert Comment

by:aolXFT
ID: 9619583
mod_rewrite isn't much good they're not using apache.
0
 
LVL 1

Expert Comment

by:fyuber
ID: 9621213
I know! But it would still be better.
0

Featured Post

Why You Should Analyze Threat Actor TTPs

After years of analyzing threat actor behavior, it’s become clear that at any given time there are specific tactics, techniques, and procedures (TTPs) that are particularly prevalent. By analyzing and understanding these TTPs, you can dramatically enhance your security program.

Join & Write a Comment

Suggested Solutions

I imagine that there are some, like me, who require a way of getting currency exchange rates for implementation in web project from time to time, so I thought I would share a solution that I have developed for this purpose. It turns out that Yaho…
Password hashing is better than message digests or encryption, and you should be using it instead of message digests or encryption.  Find out why and how in this article, which supplements the original article on PHP Client Registration, Login, Logo…
Learn how to match and substitute tagged data using PHP regular expressions. Demonstrated on Windows 7, but also applies to other operating systems. Demonstrated technique applies to PHP (all versions) and Firefox, but very similar techniques will w…
The viewer will learn how to look for a specific file type in a local or remote server directory using PHP.

743 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

12 Experts available now in Live!

Get 1:1 Help Now