Solved

script to get info over http

Posted on 2001-07-30
4
154 Views
Last Modified: 2010-03-05
anyone ever see this?

http://www.experts-exchange.com/jsp/qShow.jsp?ta=suggestion&qid=10066793


how the hell did he do it?

just give me some ideas...

seems to me he would have to search through every paq.

isn't that an insane overhead?
0
Comment
Question by:bebonham
  • 3
4 Comments
 
LVL 8

Accepted Solution

by:
shlomoy earned 100 total points
ID: 6336686
This is the way to go:

take the EE from page and extract all the urls of the various sections.

for each section url {
     get the HTML of that section
     extract all experts names from the page
     for each expert (but only for the first time you saw that exprts - as you don't want to process every expert more than once) {
          get the member profile page
          for each section in the "answered questions" section {
               save the sections's title along with the points
               be sure to get all the pages listing each section using the next XX link (this will need to be iterative - add the points until you finish all section's questions)
          }
     }
     repeat that also for the pages you can reach by following the next XX links
}



I think that's about it.
0
 
LVL 8

Author Comment

by:bebonham
ID: 6338823
thank you sir,

ouch.  That's a lot of requests...

so you are saying use LWP right?

basically, get an array like @membernames

through the process you described, and then

foreach(@membernames)
{
get "ee/jsp/memberProfile.jps?mbr=$_";
##then process
}

something like that?

I saw Interiot did a nice script for checking new questions too. You can see it on his profile.

thanks, shlomoy

regards,

Bob
0
 
LVL 8

Expert Comment

by:shlomoy
ID: 6339893
Many many requests.
You can dramatically reduce the number of requests if you have access to EE's database.

You can use LWP for doing "GET", sure :-)

You are right. You got the idea!


Sure.
Glad to help.

I'm actually very interested in such scripts which "data mine" sites.

0
 
LVL 8

Expert Comment

by:shlomoy
ID: 6339905
can you give me a link to his script?
I couldn't find it from his profile
0

Featured Post

Top 6 Sources for Identifying Threat Actor TTPs

Understanding your enemy is essential. These six sources will help you identify the most popular threat actor tactics, techniques, and procedures (TTPs).

Join & Write a Comment

There are many situations when we need to display the data in sorted order. For example: Student details by name or by rank or by total marks etc. If you are working on data driven based projects then you will use sorting techniques very frequently.…
Checking the Alert Log in AWS RDS Oracle can be a pain through their user interface.  I made a script to download the Alert Log, look for errors, and email me the trace files.  In this article I'll describe what I did and share my script.
Explain concepts important to validation of email addresses with regular expressions. Applies to most languages/tools that uses regular expressions. Consider email address RFCs: Look at HTML5 form input element (with type=email) regex pattern: T…
Get a first impression of how PRTG looks and learn how it works.   This video is a short introduction to PRTG, as an initial overview or as a quick start for new PRTG users.

744 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

11 Experts available now in Live!

Get 1:1 Help Now