<div>
Err oops submitted early. Yeah PERL = built in pattern matching, easy shuffling of large amounts of data, TONS of example code on the web dealing with Web interface
</div>


Err oops submitted early. Yeah PERL = built in pattern matching, easy shuffling of large amounts of data, TONS of example code on the web dealing with Web interface

<div>
<a href="http://search.cpan.org/~petdance/WWW-Mechanize-1.30/lib/WWW/Mechanize.pm" rel="ugc">http://search.cpan.org/~petdance/WWW-Mechanize-1.30/lib/WWW/Mechanize.pm</a><br />
<a href="http://search.cpan.org/~kventin/WWW-Robot-0.025/lib/WWW/Robot.pm" rel="ugc">http://search.cpan.org/~kventin/WWW-Robot-0.025/lib/WWW/Robot.pm</a>
</div>


http://search.cpan.org/~petdance/WWW-Mechanize-1.30/lib/WWW/Mechanize.pm [http://search.cpan.org/~petdance/WWW-Mechanize-1.30/lib/WWW/Mechanize.pm]
http://search.cpan.org/~kventin/WWW-Robot-0.025/lib/WWW/Robot.pm [http://search.cpan.org/~kventin/WWW-Robot-0.025/lib/WWW/Robot.pm]

<div>
If your budget is low then go for Open source like PHP or Java. I would recommand Java because you can avail the &nbsp;multithreading environment &nbsp;and Linux OS which is cheaper then MS Windows.<br />
<br />
if budget is not an issue then you go for MS technologies like C#.net, VB.net or VC++ .net.<br />
For MS technology I would recommand C# because easy you can avail the coding style of C++ and user interface of VB.<br />
<br />
I have developed it so far using c# when I was learner of c#.<br />
<br />
----<br />
Satyadev
</div>


If your budget is low then go for Open source like PHP or Java. I would recommand Java because you can avail the  multithreading environment  and Linux OS which is cheaper then MS Windows.

if budget is not an issue then you go for MS technologies like C#.net, VB.net or VC++ .net.
For MS technology I would recommand C# because easy you can avail the coding style of C++ and user interface of VB.

I have developed it so far using c# when I was learner of c#.

----
Satyadev

<div>
<div class="content wysiwyg-content">
Hello,<br />
<br />
We are wanting to develop a web crawler and a rather complex indexer and would like to know what language would be best suited for the task. Our aim is to crawl millions of sites and indexing based on complex pattern matching
</div>
</div>


Hello,

We are wanting to develop a web crawler and a rather complex indexer and would like to know what language would be best suited for the task. Our aim is to crawl millions of sites and indexing based on complex pattern matching

What language is the best for a web crawler and indexer ?

Web development includes all aspects of presenting content on intranets and the Internet, including delivery development, protocols, languages and standards, server software, browser clients, databases and multimedia generation.

Web Development

Programming theory deals with the design, implementation, analysis, characterization, and classification of programming languages and their individual features, along with introductory programming concepts like program structure, variable declaration and conditional and looping constructs. Sub-disciplines include the formal semantics of programming languages, type theory, program analysis and transformation, comparative programming language analysis, metaprogramming, domain-specific languages, compiler construction and run-time systems.

Programming Theory

A programming language is a formal constructed language designed to communicate instructions to a machine, particularly a computer. Thousands of different programming languages have been created, mainly in the computer field, and many more still are being created every year. The description of a programming language is usually split into the two components of syntax (form) and semantics (meaning). Some languages are defined by a specification document (for example, the C programming language is specified by an ISO Standard), while other languages (such as Perl) have a dominant implementation that is treated as a reference. Some languages have both, with the basic language defined by a standard and extensions taken from the dominant implementation being common.