Solved

how much data can mysql handle, really

Posted on 2010-11-17
11
718 Views
Last Modified: 2012-05-10
Hi

Is mysql capable of handling joins on really large tables?

I am having a problem working with large amounts of data. I tried to join a table with 1 million records to a table with 100,000 records and the join 'ate up' all the memory in the machine so that everything just died.

Mysql is an enterprise database but I have no idea if it can handle database joins on these types of scale.

Is there a way to help it out in these situations? I'm sure indexing will help but is there anyway of breaking a join down into blocks or steps

thanks
0
Comment
Question by:andieje
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 6
  • 5
11 Comments
 
LVL 143

Expert Comment

by:Guy Hengel [angelIII / a3]
ID: 34154230
table structure is?
indexes on the tables are?
query is?
explain plan of the query is?

configuration/tuning tips are here:
http://www.mysqlperformanceblog.com/2006/09/29/what-to-tune-in-mysql-server-after-installation/
0
 

Author Comment

by:andieje
ID: 34154291
I don't have a specific example, this is an exploratoty question

Simple one to many relationship
Lets say i have table A and table B. A contains a million records and table B contains a 100,000 records. There is a one to many relationship between table A and B. They have a common field which is the prinary key in table A and is a foreign key in table B. This field is indexed.

select * from table a, table b inner join on a.pk = b.fk

0
 
LVL 143

Expert Comment

by:Guy Hengel [angelIII / a3]
ID: 34154305
the correct syntax, in MySQL, would be either:
select * from table a join table b on a.pk = b.fk

Open in new window

or
select * from table a , table b where a.pk = b.fk

Open in new window


and when you say both pk and fk fields are indexes, the join should use the indexes to perform the join... eventually.
it does depend on the configuration of the memory assigned to mysql (see link above)
0
Enterprise Mobility and BYOD For Dummies

Like “For Dummies” books, you can read this in whatever order you choose and learn about mobility and BYOD; and how to put a competitive mobile infrastructure in place. Developed for SMBs and large enterprises alike, you will find helpful use cases, planning, and implementation.

 

Author Comment

by:andieje
ID: 34154308
I am asking this question because i have been told mysql won't handle the type of data I want it to handle because when someone else tried it, mysql simply 'died.' I'm hoping not to waste my time if I can avoid it

0
 
LVL 143

Expert Comment

by:Guy Hengel [angelIII / a3]
ID: 34154313
I have to presume that the "someone else" did not know about the configuration you have to do with MySQL to allow it to take more RAM than it does by default... which is usually much too low, by default ...
0
 

Author Comment

by:andieje
ID: 34154320
what do you mean the 'join should use the indexes to perform the join eventually'

I apologise for getting the syntax wrong. My question isn't about the syntax though its simple, should mysql be able to handle a simple query like that with that much data or would i really have to fine tune the configuration?
0
 
LVL 143

Accepted Solution

by:
Guy Hengel [angelIII / a3] earned 500 total points
ID: 34154321
Mysql is able to handle a join between tables having millions of records, espacially if there are appropriate indexes.
0
 

Author Comment

by:andieje
ID: 34154337
this is what i thought but i've been warned by someone not to do what I was planning to do, but as you say they might have left the innodb_buffer_pool_size at 1M

I don't suppose youve ran mysql on the cloud have you? I was wondering how much memory they would let you assign to mysql. I'm not the 'cloud person' in my team. I'm not even the computer person :)
0
 
LVL 143

Expert Comment

by:Guy Hengel [angelIII / a3]
ID: 34154368
>I don't suppose youve ran mysql on the cloud have you?
no. only a couple of distinct instances, each for a very specific task each.
the largest installation has some 200 tables, each 4GB each table (1 table = 1 day of data, actually)
0
 

Author Comment

by:andieje
ID: 34154383
you've answered many of my questions before which i really appreciate so I know you know your stuff inside out!

I would really appreciate it if you could look at this question for me if you have any time
http://www.experts-exchange.com/Database/MySQL/Q_26619565.html

many thanks
0
 

Author Closing Comment

by:andieje
ID: 34201327
thanks
0

Featured Post

Resolve Critical IT Incidents Fast

If your data, services or processes become compromised, your organization can suffer damage in just minutes and how fast you communicate during a major IT incident is everything. Learn how to immediately identify incidents & best practices to resolve them quickly and effectively.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Introduction In this installment of my SQL tidbits, I will be looking at parsing Extensible Markup Language (XML) directly passed as string parameters to MySQL 5.1.5 or higher. These would be instances where LOAD_FILE (http://dev.mysql.com/doc/refm…
Introduction This article is intended for those who are new to PHP error handling (https://www.experts-exchange.com/articles/11769/And-by-the-way-I-am-New-to-PHP.html).  It addresses one of the most common problems that plague beginning PHP develop…
Finding and deleting duplicate (picture) files can be a time consuming task. My wife and I, our three kids and their families all share one dilemma: Managing our pictures. Between desktops, laptops, phones, tablets, and cameras; over the last decade…

726 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question