Solved

script to get info over http

Posted on 2001-07-30
4
162 Views
Last Modified: 2010-03-05
anyone ever see this?

http://www.experts-exchange.com/jsp/qShow.jsp?ta=suggestion&qid=10066793


how the hell did he do it?

just give me some ideas...

seems to me he would have to search through every paq.

isn't that an insane overhead?
0
Comment
Question by:bebonham
  • 3
4 Comments
 
LVL 8

Accepted Solution

by:
shlomoy earned 100 total points
ID: 6336686
This is the way to go:

take the EE from page and extract all the urls of the various sections.

for each section url {
     get the HTML of that section
     extract all experts names from the page
     for each expert (but only for the first time you saw that exprts - as you don't want to process every expert more than once) {
          get the member profile page
          for each section in the "answered questions" section {
               save the sections's title along with the points
               be sure to get all the pages listing each section using the next XX link (this will need to be iterative - add the points until you finish all section's questions)
          }
     }
     repeat that also for the pages you can reach by following the next XX links
}



I think that's about it.
0
 
LVL 8

Author Comment

by:bebonham
ID: 6338823
thank you sir,

ouch.  That's a lot of requests...

so you are saying use LWP right?

basically, get an array like @membernames

through the process you described, and then

foreach(@membernames)
{
get "ee/jsp/memberProfile.jps?mbr=$_";
##then process
}

something like that?

I saw Interiot did a nice script for checking new questions too. You can see it on his profile.

thanks, shlomoy

regards,

Bob
0
 
LVL 8

Expert Comment

by:shlomoy
ID: 6339893
Many many requests.
You can dramatically reduce the number of requests if you have access to EE's database.

You can use LWP for doing "GET", sure :-)

You are right. You got the idea!


Sure.
Glad to help.

I'm actually very interested in such scripts which "data mine" sites.

0
 
LVL 8

Expert Comment

by:shlomoy
ID: 6339905
can you give me a link to his script?
I couldn't find it from his profile
0

Featured Post

Is Your Active Directory as Secure as You Think?

More than 75% of all records are compromised because of the loss or theft of a privileged credential. Experts have been exploring Active Directory infrastructure to identify key threats and establish best practices for keeping data safe. Attend this month’s webinar to learn more.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
Which version of C should I use when Perl is too slow? 13 172
compress files in RAR using perl 13 79
Port 80 requests 16 86
Perl Awk Need Help 3 109
Email validation in proper way is  very important validation required in any web pages. This code is self explainable except that Regular Expression which I used for pattern matching. I originally published as a thread on my website : http://www…
There are many situations when we need to display the data in sorted order. For example: Student details by name or by rank or by total marks etc. If you are working on data driven based projects then you will use sorting techniques very frequently.…
Explain concepts important to validation of email addresses with regular expressions. Applies to most languages/tools that uses regular expressions. Consider email address RFCs: Look at HTML5 form input element (with type=email) regex pattern: T…
Need to grow your business through quality cloud solutions? With everything required to build a cloud platform and solution, you may feel like the distance between you and the cloud is quite long. Help is here. Spend some time learning about the Con…

943 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

11 Experts available now in Live!

Get 1:1 Help Now