• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 205
  • Last Modified:

script to get info over http

anyone ever see this?

http://www.experts-exchange.com/jsp/qShow.jsp?ta=suggestion&qid=10066793


how the hell did he do it?

just give me some ideas...

seems to me he would have to search through every paq.

isn't that an insane overhead?
0
bebonham
Asked:
bebonham
  • 3
1 Solution
 
shlomoyCommented:
This is the way to go:

take the EE from page and extract all the urls of the various sections.

for each section url {
     get the HTML of that section
     extract all experts names from the page
     for each expert (but only for the first time you saw that exprts - as you don't want to process every expert more than once) {
          get the member profile page
          for each section in the "answered questions" section {
               save the sections's title along with the points
               be sure to get all the pages listing each section using the next XX link (this will need to be iterative - add the points until you finish all section's questions)
          }
     }
     repeat that also for the pages you can reach by following the next XX links
}



I think that's about it.
0
 
bebonhamAuthor Commented:
thank you sir,

ouch.  That's a lot of requests...

so you are saying use LWP right?

basically, get an array like @membernames

through the process you described, and then

foreach(@membernames)
{
get "ee/jsp/memberProfile.jps?mbr=$_";
##then process
}

something like that?

I saw Interiot did a nice script for checking new questions too. You can see it on his profile.

thanks, shlomoy

regards,

Bob
0
 
shlomoyCommented:
Many many requests.
You can dramatically reduce the number of requests if you have access to EE's database.

You can use LWP for doing "GET", sure :-)

You are right. You got the idea!


Sure.
Glad to help.

I'm actually very interested in such scripts which "data mine" sites.

0
 
shlomoyCommented:
can you give me a link to his script?
I couldn't find it from his profile
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

Featured Post

Cloud Class® Course: Amazon Web Services - Basic

Are you thinking about creating an Amazon Web Services account for your business? Not sure where to start? In this course you’ll get an overview of the history of AWS and take a tour of their user interface.

  • 3
Tackle projects and never again get stuck behind a technical roadblock.
Join Now