Solved

RSS html rip

Posted on 2006-11-28
5
449 Views
Last Modified: 2012-06-27
Hi

I want to make an RSS feed.  What I want it to do is first log into a site, get values from a specific HTML page, and then put these values into the feed.  Using PHP, Perl, Ruby or Java, does anyone know a way that I can fetch a HTML page from an external web site (remembering that it will need to provide some authentication first)?

Specifically what I'm looking for is a way to fetch a page behind authentication automatically, using my username and password for the site, and to basically save that page to my server so that I can work on pulling out the values I want.
0
Comment
Question by:Mr_Lenehan
  • 3
  • 2
5 Comments
 
LVL 2

Author Comment

by:Mr_Lenehan
ID: 18032935
also, it could use any unix/linux program such as wget as this could be set as this could be activated from within the coded solution.
0
 
LVL 5

Expert Comment

by:dasmaer
ID: 18033040
wget can do it...
   
   wget --help

you'll see it can do HTTP authentication, and also cookie loading - so you'd need to find your cookie placed by the site at log in.
0
 
LVL 2

Author Comment

by:Mr_Lenehan
ID: 18033144
I tried that... I pointed wget at the location of the cookie and it didn't work! (the site doesn't use HTTP authentication)

Maybe there's a different way? programmatically?
0
 
LVL 5

Accepted Solution

by:
dasmaer earned 500 total points
ID: 18033157
using cookies has nothing to do with HTTP auth.

Try this, forcing wget to use the cookie you want. Specify cookies off and then the cookie name and value (see: http://www.delorie.com/gnu/docs/wget/wget_9.html).  

Example: wget --cookies=off --header="Cookie: LOGIN=username:passwordhash" http://www.yoursite.com

where password hash is either a plain text password or the hashed password in your cookie.
0
 
LVL 2

Author Comment

by:Mr_Lenehan
ID: 18033172
Result! Using php shell_exec I can get this to fetch my page (well, not my page but that page I want). Cheers.
0

Featured Post

Highfive + Dolby Voice = No More Audio Complaints!

Poor audio quality is one of the top reasons people don’t use video conferencing. Get the crispest, clearest audio powered by Dolby Voice in every meeting. Highfive and Dolby Voice deliver the best video conferencing and audio experience for every meeting and every room.

Join & Write a Comment

It's sometimes a bit tricky to use date functions in Oracle BPEL. I'll explain quickly how you can add N days to the current date. In a BPEL process this can be useful, and you can adapt it to fit your needs. First of all, let's see how to add 1 …
Styling your websites can become very complex. Here I'll show how SASS can help you better organize, maintain and reuse your CSS code.
The viewer will learn how to look for a specific file type in a local or remote server directory using PHP.
The viewer will the learn the benefit of plain text editors and code an HTML5 based template for use in further tutorials.

708 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

14 Experts available now in Live!

Get 1:1 Help Now