Solved

NoSQL or RDBMS? Web crawler

Posted on 2012-03-27
3
1,137 Views
Last Modified: 2016-02-10
I need to crawl selected websites for about 5-10 different attributes.

For example lets say its a car website and on each page of the site there is information about a particular car for sale and it includes the vehicle make, model, year, price and etc.
I need all this information to be collected and stored in a database but since a good car sale website could have thousands of pages it can become a lot of data to collect.

I don't expect to have more than a few hundred words collected from each page so i think it would be under 1KB of data per record i store.

At the moment I don't know if i should be using NoSQL or a MySQL database since i will have an insane amount of rows/records created.

Any thoughts on going one way or the other? I need to do certain data manipulation on all the rows/records such as organizing the car by price from highest to lowest and etc.
0
Comment
Question by:checkmofoshoduno
3 Comments
 
LVL 24

Accepted Solution

by:
johanntagle earned 500 total points
ID: 37774846
Either MySQL or MongoDB (that's the only NoSQL db I've used so far) should be able to handle your needs, though MongoDB provides better horizontal scaling via sharding, should your data be really that huge.  If everything can be contained on one server, I would think the decision would depend on whether or not you can pre-define all the fields you need.  If so, I would personally use MySQL because I find querying via SQL more straightforward vs having to deal with mapreduce and the like.  But if you need to dynamically store different field names, then a NoSQL database is the way to go.
0

Featured Post

U.S. Department of Agriculture and Acronis Access

With the new era of mobile computing, smartphones and tablets, wireless communications and cloud services, the USDA sought to take advantage of a mobilized workforce and the blurring lines between personal and corporate computing resources.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

This is about my first experience with programming Arduino.
Computer science students often experience many of the same frustrations when going through their engineering courses. This article presents seven tips I found useful when completing a bachelors and masters degree in computing which I believe may he…
With the power of JIRA, there's an unlimited number of ways you can customize it, use it and benefit from it. With that in mind, there's bound to be things that I wasn't able to cover in this course. With this summary we'll look at some places to go…

860 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question