Still celebrating National IT Professionals Day with 3 months of free Premium Membership. Use Code ITDAY17

x

NoSQL Databases

124

Solutions

242

Contributors

A NoSQL database provides a mechanism for storage and retrieval of data which is modeled in means other than the tabular relations used in relational databases. Motivations for this approach include: simplicity of design, simpler "horizontal" scaling to clusters of machines and finer control over availability. The data structures used by NoSQL databases (e.g. key-value, wide column, graph, or document) are specified from those used by default in relational databases, making some operations faster in NoSQL. Sometimes the data structures used by NoSQL databases are also viewed as "more flexible" than relational database tables.

Share tech news, updates, or what's on your mind.

Sign up to Post

When I want to sort by multiple keys on mongodb , I got some problem , And can I fix it ??

db.test.a.count()

78150

db.test.a.ensureIndex({a:1,b:1,c:1,d:1});

db.test.a.find().sort({a:1,b:1,c:1,d:1});

error: { "$err" : "too much data for sort() with no index. add an index or speci fy a smaller limit", "code" : 10128 }
0
Hi,

There are records in one of my Mongodb collection 'exmple' (please see attached code).

I would like to have Java queries so that I can get the records for each of the following scenarios:

1) records whose "nextfetchtime" field is not exist;

2) records whose "nextfetchtime" field has value of null;

Thanks.
{ "_id" : { "$oid" : "4e72eac3e4b0a2c48b0eb84d"} , "code" : "5g9a4gini94404mp1inoip5ni4gpcgn4gnin50i0ci5a0n95g555o4pgg1m45o15" , "path" : "/data2/data/htmlsource/2011-09-16/dianpingcom/" , "finishtime" : 1316154241099 , "nextfetchtime" : 1318746051658 , "codeTime" : 1316154051696 , "predicted" : false}
{ "_id" : { "$oid" : "4e7564247af2f2aa38b3fbe8"} , "code" : "mw2Da0toDk6xhkz6IVNxmEQSsQ3nLnJ84Ccxvw0734i5Hv5auTHk5aQtmUq1KxhY" , "path" : "c:/data/berkeley/database/parsedcodedb-bak/2011-07-03/dianping" , "finishtime" : 1316316196568}
{ "_id" : { "$oid" : "4e7564377af2587c0e06e5df"} , "code" : "fl6Q3Nw3BCMrX5LxHoY49llQ81vfYtW0yTGec9arAHYch8yvDvGPJ37LOzKwOOgc" , "path" : "c:/data/berkeley/database/parsedcodedb-bak/2011-07-03/dianping" , "finishtime" : 1316316215222}
{ "_id" : { "$oid" : "4e7568e07af237357a034671"} , "code" : "QfDfFpWqBHAtJYkJbMdTWFl5UrLx6lp96bdOFGzIOs0m1RlqRPxhIC0RA9ONB2yh" , "path" : "c:/data/berkeley/database/parsedcodedb-bak/2011-07-03/dianping" , "finishtime" : 1316317408821 , "nextfetchtime" :  null }

Open in new window

0
Hi,

I needs to handle millions of json files, which involves multiple steps, and saving a copy of handled files at each step.

Originally I wanted to save those files in FS, but now I am thinking of MongoDB. I would like to know what differences, in terms of spead, are.

Thanks
0
Hi,

I have known mysql replication, but I want to know what are the tools to replicate file systems (lucene, mongodb, etc.).

Please advice!

Thanks
0
Hi,

Currently, we are using mysql and Hadoop for our distributed computing solution. The architecture looks like this: (http://zcitrq.bay.livefilestore.com/y1p82no83SaHmBi7CfT7zhR3Fg83JdGxadCIK54-SDzk1I-8jRhFtE53sGcw0IUeOYmAR29C6GgzuNY6Vo8jZeH8PaRz97VX8NL/mapreduce.png?psid=1)

The issue, however, is that connecting to mysql turns out to be very expensive. I want to know if there are other database solutions that work well with Hadoop.

The solution should do the following job:

1. Both reading and writing to the database (distributed computing friendly) are needed;
2. Both Namenode and datanodes can read and write to the database;
3. Seamlessly working with Hadoop HDFS is preferred, if possible.

Thanks
0
Hi, I have a mongodb with image data stored in it's table in the unique MongoDb way. I'm assuming its what blob is to MySQL. So what are the statements i would use to retrieve an Image from its database and display it on a page? I've provided a VERY SIMPLE example of what I would do if it were MySQL so please now show me how to do it with php and MongoDB. Thanks in advance experts!
<?php

//view.php
//This is to display the image
//get the image from the db
$sql = 'SELECT image FROM image_table WHERE image_id = "'.$_GET['image_id'].'"';

//the result of the query
$result = mysql_query($sql);

//set the header for the image
header("Content-type: image/jpeg");
echo mysql_result($result,0);
?>

<?php

//gallery.php
//This is to display the gallery

//get the images
$sql = 'SELECT * FROM image_table WHERE';

//the result of the query
$result = mysql_query($sql);

//set the img tags
while($row = mysql_fetch_array($result){
    echo '<img src="view.php?image_id='.$row['image_id'].'" />';
}
?>

Open in new window

0
I'm stuck and I NEED the mongo drivers installed. I'm a reseller so I have WHM access and cpanel access to the server as well, so how do I install the mongo drivers?
0
We are looking at developing a cloud (probably Amazon EC2) based application that might potentially scale to tens of thousands of small transactions per second via a RESTful API. The architecture we are looking at is to allow our client facing application to put transactions into a high performance message queue from which they will be processed asynchronously by background worker agents and logged into a database. The client facing application will probably be based on Ruby on Rails, while the backend will either be Ruby or Java.

We have been looking at using some of the EC2 services such as SQS, SimpleDB as basic building blocks as we were attracted by the scalability and performance promised by Amazon. We've done some initial testing and find that throughput performance is very poor. For example with SQS we've only been able to get around 1-2 transactions per second. Our initial testing shows we may have similar issues with SimpleDB.

We're now looking at some other options, including using an AMQP standards based messaging tools such as RabbitMQ. While using MySQL as a backend db is attractive we're also considering using a nosql solution such as MongoDB.

We'd be interested in any any ideas or feedback on the approach we are taking. We would also like to learn if anyone has any experience running an architecture like this in the cloud (in particular EC2).
0
Hi experts,

I was wondering if any of you know what database does twitter use?

I read this article but I still don't get an answer.

http://ubuntuforums.org/showthread.php?t=1294882

Since twitter gets so much daily traffic I just wanted to know what database they use.

I know Google uses their own custom in-house solution they created from scratch, instead of storing their data in SqlServer, Oracle or Terradata,  so I was wondering if maybe Twitter did the same or do they use Oracle or Terradata for their data storage so they could be able to serve up so many daily transactions.

Thanks for any help you can give me on this question.
0
I am trying to install the Perl driver for MongoDB on my CentOS 5 64 bit machine.

I found an RPM here:
http://rpmfind.net/linux/rpm2html/search.php?query=perl(mongodb)&submit=Search+...&system=&arch=

But the RPM was for Fedora, not CentOS.

I tried it anyway, like this:

   wget ftp://rpmfind.net/linux/fedora/development/rawhide/x86_64/os/Packages/perl-MongoDB-0.41-3.fc16.x86_64.rpm
    rpm -Uvh perl-MongoDB-0.41-3.fc16.x86_64.rpm


These errors were returned:
       error: Failed dependencies:
        perl(:MODULE_COMPAT_5.12.3) is needed by perl-MongoDB-0.41-3.fc16.x86_64
        rpmlib(FileDigests) <= 4.6.0-1 is needed by perl-MongoDB-0.41-3.fc16.x86_64
        perl(Any::Moose) is needed by perl-MongoDB-0.41-3.fc16.x86_64
        perl(boolean) is needed by perl-MongoDB-0.41-3.fc16.x86_64
        perl(DateTime) is needed by perl-MongoDB-0.41-3.fc16.x86_64
        perl(Tie::IxHash) is needed by perl-MongoDB-0.41-3.fc16.x86_64
        rpmlib(PayloadIsXz) <= 5.2-1 is needed by perl-MongoDB-0.41-3.fc16.x86_64

Is there any way I can make this Fedora RPM work on my CentOS 5 machine?
0
I'm a reasonably decent PHP programmer but I just don't "get" the object orientated side and it's high time I sorted it out. This may not be worth 500 points but I thought I'd try anyway.

In the code snippet I have some code I've written to pull some info from a mongoDB. I realised I've used some classes in there but I haven't constructed any - they're just what the MongoDB driver provides.
In the foreach loop I have some variables that are printed out from the MongoDB. I'd like to make that foreach loop a class so that I can call something like:

print->Environment
print->Architecture

which makes it nice and clean and simple. I figure if someone can help me create a class from something I've already written it might help me "get" classes and objects. All help gratefully considered.
$gethost=$_GET['q'];
try {
  // open connection to MongoDB server
  $conn = new Mongo('cbb-cs03-bt26-17');

  // access database
  $db = $conn->factdb;

  //authenticate
  $db->authenticate('username','password');

  // access collection
  $collection = $db->hosts;

  // define what to find
  $host = array(
        'host' => $gethost
);
  // disconnect from server
  $conn->close();
} catch (MongoConnectionException $e) {
  die('Error connecting to MongoDB server');
} catch (MongoException $e) {
  die('Error: ' . $e->getMessage());
}


 $cursor = $collection->find($host);
  foreach ($cursor as $value) {
          echo 

Open in new window

0
Hi
I have installed nagios 3.2.2 and the plug-ins check_mongo.py for some reason the plug-ins show some server in critical state with this message: CRITICAL - Connection to MongoDB failed!
its in a secure env so no authentication, I ran each check command that i have in my nrpe,cfg i.e:
command[check_mongo_connect]=/usr/local/nagios/libexec/check_mongodb.py -A connect -W 2 -C 4
command[check_mongo_free]=/usr/local/nagios/libexec/check_mongodb.py -A connections -W 70 -C 80
command[check_mongo_rep]=/usr/local/nagios/libexec/check_mongodb.py -A check_rep_lag -W 2 -C 80

on the client and the nagios host (with -H) it all returned connection ok but still nagios show them as critical, nrpe is running in xinetd.
Thanks
0
I have an existing array that needs to be converted into a table of it's own in a database. Can anyone help me find or create a function in php that could do this?

But then I also need another function that would easily read the new table back into an array so I don't have to change a large section of my existing code.

Any help would be appreciated.
0
hi guys

I wanted to gain some knowledge on NOSQL databases like google's bigtable and couchdb,mongodb,ect
They are gaining extreme popularity due to their speed and performance.
Does anyone have any practical experiences to share.
0
Suppose there is highly interactive and very high traffic website (10,000 visitors per day) with interactive content and gaming applications that requires each game move to be persisted to datastore along with over a million user profiles. Think of games like farmville etc.

The requirement is that datastore must be based on one of NoSQL datastores. No relational database like MySQL can be used in any place in the whole web application.

Extremely high performance like super fast writes and reads is number one requirement.
Second most important requirement is data security and automatic fail over in case one part of datastore gets corrupted, another takes over.

Third requirement it should be open source and free for commercial use.
Fourth requirement good tools should be avaialble to admin and query the datastore during development time like we have tools MySQLWorkbench for MySQL database.


I notice a whole bunch alternative in the NoSQL space like:
Membased, Cassandra, Redis, Hadoop, BigTable etc.

From your experience what do you  recommend and why given the above requirements?
0
Ok, here's the scenario.

I do work for a lead processing company.  This company is starting a new program where they will be processing leads from many sources, and distributing to many other sources.  I've been tasked with setting up a system to recieve these leads and store them in a db.  There will be a web site built for retrieving data in many formats, as well as putting data into the db.

Here's my question.....
Having setup similar systems in the past, i've found that different sources have different ways of doing things.  Their data will never be in the same formats.  They always want to retrieve information in different file formats as well.

When designing this table...which will have about 200+ fields... some of which will be used, some won't depending on who is sending and recieving and what they need....  should I just make all the fields Varchar ??    I mean, i've found that having proper formatted fields works great when you are building an application that needs validation, and the data is the same everytimme, but i'm thinking in this scenario...if I just treat all data as text, i'll never have a read/write failure to the db.  The web code that will recieve and deliver the data would be simplified without having to validate field data.  The onus would be on the source, to make sure what they put in is what they want out.  I'm also thinking.... if everything is varchar, then it's very easy to convert in code to any datatype if ever I need to perform …
0
Dear EE Members,

I am exploring the possibility of doing a SaaS based computing platform similar to those of salesforce.com, zoho, and the likes,

What we need to decide is which database platform to use i.e. MySQL, MS-SQL, Oracle, or any other format,

Any insight, on what salesforc, zoho, googleapps, facebook uses, would be great we are looking for some concrete advise, so we can decide at the design stage.
0
Somebody know how to use session variables between servers????

for example:

In localhost I had sesion.php with this code:
<?
session_start();
?>
<html>
<head>
<title>Generar variable de sesión</title>
</head>
<body>

<form action="http://www.itssc.edu.mx/sesion3.php" method="POST">
<p>Id
<input type="text" name="id" value="" size="">
</p>
<p>
Nombre
<input type="text" name="nombre" value="" size="">
</p>
<p>
<input name="enviar" type="submit" value="CONTINUAR">
</p>
</form>
</body>
</html>
--------------------------------------
I want to use the value in the variables inside sesion.php in http://www.itssc.edu.mx/sesion3.php

<?
session_start();
?>
<?
$_SESSION['direccion']=$_POST[direccion];
?>
<html>
<head>
<title>Muestra las variables de sesi&ocuate;n</title>
</head>
<body>
Muestro las variables:
<?
echo "<br>";
echo $_SESSION['id'];
echo "<br>";
echo $_SESSION['nombre'];
echo "<br>";

?>
</body>
</html>
----------------------------------------------------------------------------

I this possible????

Can I use session variables with php pages between diferent servers?????
0
Can someone help me understand this code.  It is meant to be an example of how to use a C# library talking to a database called Mongo DB (http://wiki.github.com/atheken/NoRM/)

When compiling I get

Error      1      The type or namespace name 'Product' could not be found (are you missing a using directive or an assembly reference?)      ...
Error      2      The type or namespace name 'Product' could not be found (are you missing a using directive or an assembly reference?)      ...
Error      3      'Norm.ObjectId' does not contain a definition for 'NewObjectID'      ...
Error      4      The type arguments for method 'Norm.Collections.MongoCollection<Product>.FindOne<U>(U)' cannot be inferred from the usage. Try specifying the type arguments explicitly.      .....

I don't understand the syntax around the <Product> - looks like a template or something.
Assuming this looks right, where should the Product template/object be defined?

Thanks
BT
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using Norm;

namespace ConsoleApplication1
{
    class Program
    {
        static void Main(string[] args)
        {
            var db = new Mongo("test", "10.1.1.20", "27017", "");
            var coll = db.GetCollection<Product>();
            //create a new object to be added to the collection
            var obj = new Product();
            obj._id = ObjectId.NewObjectID();
            obj.Title = "Shoes";
            //save the object
            coll.Insert(obj);
           

Open in new window

0
Hi,

Could someone explain what is going on here please.  I'm using C# and Visual Studio 2008

I am trying to use a C# DLL library for a system called Redis (http://code.google.com/p/servicestack/wiki/ServiceStackRedis).  This DLL allows me to send messages to and from Redis data store.

My first test was to create a Windows Console application to which I was able to add the DLL to the project references by selection Add Reference and being able to BROWSE to the DLL from the Add Ref dialog.
This linked the DLL to the project and I was able to use the functions within.

However, my end goal is to use the DLL in a Database | SQL Server Project to create a CLR DLL to be added to SQL Server as an Assembly.
When I create such a project and then try to Add References, the dialog only has tabs for Projects and SQL Server, but no tab for browsing to the Redis DLL.

Is there a way to add the Redis DLL to the SQL Server project?

Thanks
BT
0
We are looking for a product (or developer) to allow us to write data directly to a Redis message store (http://code.google.com/p/redis).
If you not familiar with Redis, the communication would be sending commands/data Redis via a IP:Port transport.

Could someone outline the options for:
1. The tools/techniques for creating such plugins for SQL Server
2. Any suitable plugins or technologies that could help

Ultimately we are looking to have a trigger call a stored procedure, that would when collect data together and send to Redis via the IP transport.

Many thanks
BT
0
I would like to keep a 'click-path' record, so that I can analyze what a particular user likes, and customize the site to their preferences.

Requirements:
1. Insert rate of about 1000 rows per second
2. Efficient key based record retrieval for both off-line analysis, and real-time reads.

Besides my requirements, I'm very interested in:
stability
ease of setup
size of community
amount of active development

Lastly, if there are any data analysis tools that specifically support the db.


*PLEASE make any recommendations on following:
+Apache CouchDB
+Apache Cassandra
+Hypertable
+Apache HBase / Hadoop
+Apache Jackrabbit
+MongoDB
0
I want to use something cheaper than MS SQL Server as a database to handle what may turn into a fairly large volume of traffic.  Basically, my app will behave the same as a Twitter feed.  People submit messages to our server and receive messages based on a query.  I'd like new messages to appear fairly quickly (few seconds).

Using MySQL or some similar database will allow me to move across hosting providers easily and without the expense of a Windows Server or MS SQL Server.  Does MySQL handle large volumes of traffic well?

Does connecting to it through PHP perform well?
0
Hi, I am looking for a newsgroup server like Cassandra by atrium that is easy to setup and maintain. Cassandra looks like it has all the features I need but I just don't get very good support from them. I need to have newsgroups that are open and public as well as groups that are private and require a valid login. We will not "suck" any feeds from other groups as we want only our specific groups. Any suggestions? Anyone who can advise on Cassandra. Thank you.
0

NoSQL Databases

124

Solutions

242

Contributors

A NoSQL database provides a mechanism for storage and retrieval of data which is modeled in means other than the tabular relations used in relational databases. Motivations for this approach include: simplicity of design, simpler "horizontal" scaling to clusters of machines and finer control over availability. The data structures used by NoSQL databases (e.g. key-value, wide column, graph, or document) are specified from those used by default in relational databases, making some operations faster in NoSQL. Sometimes the data structures used by NoSQL databases are also viewed as "more flexible" than relational database tables.

Top Experts In
NoSQL Databases
<
Monthly
>