Want to win a PS4? Go Premium and enter to win our High-Tech Treats giveaway. Enter to Win


RSS html rip

Posted on 2006-11-28
Medium Priority
Last Modified: 2012-06-27

I want to make an RSS feed.  What I want it to do is first log into a site, get values from a specific HTML page, and then put these values into the feed.  Using PHP, Perl, Ruby or Java, does anyone know a way that I can fetch a HTML page from an external web site (remembering that it will need to provide some authentication first)?

Specifically what I'm looking for is a way to fetch a page behind authentication automatically, using my username and password for the site, and to basically save that page to my server so that I can work on pulling out the values I want.
Question by:Mr_Lenehan
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 3
  • 2

Author Comment

ID: 18032935
also, it could use any unix/linux program such as wget as this could be set as this could be activated from within the coded solution.

Expert Comment

ID: 18033040
wget can do it...
   wget --help

you'll see it can do HTTP authentication, and also cookie loading - so you'd need to find your cookie placed by the site at log in.

Author Comment

ID: 18033144
I tried that... I pointed wget at the location of the cookie and it didn't work! (the site doesn't use HTTP authentication)

Maybe there's a different way? programmatically?

Accepted Solution

dasmaer earned 2000 total points
ID: 18033157
using cookies has nothing to do with HTTP auth.

Try this, forcing wget to use the cookie you want. Specify cookies off and then the cookie name and value (see: http://www.delorie.com/gnu/docs/wget/wget_9.html).  

Example: wget --cookies=off --header="Cookie: LOGIN=username:passwordhash" http://www.yoursite.com

where password hash is either a plain text password or the hashed password in your cookie.

Author Comment

ID: 18033172
Result! Using php shell_exec I can get this to fetch my page (well, not my page but that page I want). Cheers.

Featured Post

Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

I will show you how to create a ASP.NET Captcha control without using any HTTP HANDELRS or what so ever. you can easily plug it into your web pages. For Example a = 2 + 3 (where 2 and 3 are 2 random numbers) Session("Answer") = 5 then we…
SASS allows you to treat your CSS code in a more OOP way. Let's have a look on how you can structure your code in order for it to be easily maintained and reused.
The viewer will learn how to dynamically set the form action using jQuery.
The viewer will learn how to create a basic form using some HTML5 and PHP for later processing. Set up your basic HTML file. Open your form tag and set the method and action attributes.: (CODE) Set up your first few inputs one for the name and …
Suggested Courses

618 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question