Solved

c# spider the web

Posted on 2010-11-26
5
455 Views
Last Modified: 2012-05-10
may programs can extract email from url and many pro gram can copy con tent of url to local machine   How can those program known the directory of url they follow the link or they have a class to explore web directory
0
Comment
Question by:teera
  • 4
5 Comments
 
LVL 10

Expert Comment

by:aboo_s
ID: 34220696
If I understand your question is how to explore all url's on the web, you can create a program that will
try all combinations in .com then .net and so on to discover which are real url by connecting to them using port 80.  you start like this a.com b.com c.com ...aa.com ab.com ...and so on ..you can explore most of the web ..!!


Does this answer your question!?
0
 

Author Comment

by:teera
ID: 34221409
Hi aboo_s

i wanto explore a.com , a.om/s.htm  a.com/sys.php a.com/pic/in.jpg  i want to explore on the url
0
 
LVL 10

Accepted Solution

by:
aboo_s earned 500 total points
ID: 34222518
Oh I see,

When you have the correct URL, let's say a.com exists, then you can list the whole directory underneath
using php command for example:  readdir (example code attached)

This way when a URL is found you can get the list of files(directories) under it, and work with them accordingly!
/* List all files in a directory */
<?php

if ($handle = opendir('/path/to/files')) {
    echo "Directory handle: $handle\n";
    echo "Files:\n";

   
    while (false !== ($file = readdir($handle))) {
        echo "$file\n";
    }

   

    closedir($handle);
}
?>

Open in new window

0
 
LVL 10

Expert Comment

by:aboo_s
ID: 34222523
Oh ..but you must ask this!!! You can only run php on the server which you cannot!!!
So ..I'll think of something ..
0
 
LVL 10

Expert Comment

by:aboo_s
ID: 34222559
Ok, I guess you will most probably have to also guess them just like with  the url s
Cause som(most) site hide their files from being listed!

Anyway because you do not have to re-invent the wheel !!!
Take a look at this page:
http://allseeing-i.com/ASIHTTPRequest/
0

Featured Post

Best Practices: Disaster Recovery Testing

Besides backup, any IT division should have a disaster recovery plan. You will find a few tips below relating to the development of such a plan and to what issues one should pay special attention in the course of backup planning.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
WKHTMLTOPDF - --disable-smart-shrinking not working 10 90
Help with C#, MVC, razor. 6 34
C# parent child form 5 28
Windows Media Player 2 26
Introduction Although it is an old technology, serial ports are still being used by many hardware manufacturers. If you develop applications in C#, Microsoft .NET framework has SerialPort class to communicate with the serial ports.  I needed to…
This article introduced a TextBox that supports transparent background.   Introduction TextBox is the most widely used control component in GUI design. Most GUI controls do not support transparent background and more or less do not have the…
Microsoft Active Directory, the widely used IT infrastructure, is known for its high risk of credential theft. The best way to test your Active Directory’s vulnerabilities to pass-the-ticket, pass-the-hash, privilege escalation, and malware attacks …
Although Jacob Bernoulli (1654-1705) has been credited as the creator of "Binomial Distribution Table", Gottfried Leibniz (1646-1716) did his dissertation on the subject in 1666; Leibniz you may recall is the co-inventor of "Calculus" and beat Isaac…

778 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question