Solved

NoSQL or RDBMS? Web crawler

Posted on 2012-03-27
3
1,103 Views
Last Modified: 2016-02-10
I need to crawl selected websites for about 5-10 different attributes.

For example lets say its a car website and on each page of the site there is information about a particular car for sale and it includes the vehicle make, model, year, price and etc.
I need all this information to be collected and stored in a database but since a good car sale website could have thousands of pages it can become a lot of data to collect.

I don't expect to have more than a few hundred words collected from each page so i think it would be under 1KB of data per record i store.

At the moment I don't know if i should be using NoSQL or a MySQL database since i will have an insane amount of rows/records created.

Any thoughts on going one way or the other? I need to do certain data manipulation on all the rows/records such as organizing the car by price from highest to lowest and etc.
0
Comment
Question by:checkmofoshoduno
3 Comments
 
LVL 24

Accepted Solution

by:
johanntagle earned 500 total points
ID: 37774846
Either MySQL or MongoDB (that's the only NoSQL db I've used so far) should be able to handle your needs, though MongoDB provides better horizontal scaling via sharding, should your data be really that huge.  If everything can be contained on one server, I would think the decision would depend on whether or not you can pre-define all the fields you need.  If so, I would personally use MySQL because I find querying via SQL more straightforward vs having to deal with mapreduce and the like.  But if you need to dynamically store different field names, then a NoSQL database is the way to go.
0

Featured Post

Comprehensive Backup Solutions for Microsoft

Acronis protects the complete Microsoft technology stack: Windows Server, Windows PC, laptop and Surface data; Microsoft business applications; Microsoft Hyper-V; Azure VMs; Microsoft Windows Server 2016; Microsoft Exchange 2016 and SQL Server 2016.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
xampp tool 12 48
DBF to ... Converter 5 69
PHP: Insert Data into MySQL 5 39
add and run outdated exchange database to an Exchange server 1 38
This is about my first experience with programming Arduino.
As technology users and professionals, we’re always learning. Our universal interest in advancing our knowledge of the trade is unmatched by most industries. It’s a curiosity that makes sense, given the climate of change. Within that, there lies a…
In this fourth video of the Xpdf series, we discuss and demonstrate the PDFinfo utility, which retrieves the contents of a PDF's Info Dictionary, as well as some other information, including the page count. We show how to isolate the page count in a…
In this fifth video of the Xpdf series, we discuss and demonstrate the PDFdetach utility, which is able to list and, more importantly, extract attachments that are embedded in PDF files. It does this via a command line interface, making it suitable …

776 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question