The entire dmoz directory in RDF format can be downloaded here:
I downloaded the entire thing. It is compressed to about 300mb but when I extract it it is actually about 1.8 gigabytes.
I want a PHP script to place every single URL in a mySQL database like this:
$query = "INSERT INTO URL VALUES ('http://www.example.com'
The problem is that I am worried that this could kill my poor server by hogging all its resources.
I would like your ideas on how this could be done in such a way as to preserve system resources. I don't care if it takes two weeks to completely place all URLs in the database.
Also, please tell me if you think another language such as PERL or a linux sh script would be a better choice for this than PHP.