Solved

how much data can mysql handle, really

Posted on 2010-11-17
11
714 Views
Last Modified: 2012-05-10
Hi

Is mysql capable of handling joins on really large tables?

I am having a problem working with large amounts of data. I tried to join a table with 1 million records to a table with 100,000 records and the join 'ate up' all the memory in the machine so that everything just died.

Mysql is an enterprise database but I have no idea if it can handle database joins on these types of scale.

Is there a way to help it out in these situations? I'm sure indexing will help but is there anyway of breaking a join down into blocks or steps

thanks
0
Comment
Question by:andieje
  • 6
  • 5
11 Comments
 
LVL 142

Expert Comment

by:Guy Hengel [angelIII / a3]
ID: 34154230
table structure is?
indexes on the tables are?
query is?
explain plan of the query is?

configuration/tuning tips are here:
http://www.mysqlperformanceblog.com/2006/09/29/what-to-tune-in-mysql-server-after-installation/
0
 

Author Comment

by:andieje
ID: 34154291
I don't have a specific example, this is an exploratoty question

Simple one to many relationship
Lets say i have table A and table B. A contains a million records and table B contains a 100,000 records. There is a one to many relationship between table A and B. They have a common field which is the prinary key in table A and is a foreign key in table B. This field is indexed.

select * from table a, table b inner join on a.pk = b.fk

0
 
LVL 142

Expert Comment

by:Guy Hengel [angelIII / a3]
ID: 34154305
the correct syntax, in MySQL, would be either:
select * from table a join table b on a.pk = b.fk

Open in new window

or
select * from table a , table b where a.pk = b.fk

Open in new window


and when you say both pk and fk fields are indexes, the join should use the indexes to perform the join... eventually.
it does depend on the configuration of the memory assigned to mysql (see link above)
0
 

Author Comment

by:andieje
ID: 34154308
I am asking this question because i have been told mysql won't handle the type of data I want it to handle because when someone else tried it, mysql simply 'died.' I'm hoping not to waste my time if I can avoid it

0
 
LVL 142

Expert Comment

by:Guy Hengel [angelIII / a3]
ID: 34154313
I have to presume that the "someone else" did not know about the configuration you have to do with MySQL to allow it to take more RAM than it does by default... which is usually much too low, by default ...
0
Control application downtime with dependency maps

Visualize the interdependencies between application components better with Applications Manager's automated application discovery and dependency mapping feature. Resolve performance issues faster by quickly isolating problematic components.

 

Author Comment

by:andieje
ID: 34154320
what do you mean the 'join should use the indexes to perform the join eventually'

I apologise for getting the syntax wrong. My question isn't about the syntax though its simple, should mysql be able to handle a simple query like that with that much data or would i really have to fine tune the configuration?
0
 
LVL 142

Accepted Solution

by:
Guy Hengel [angelIII / a3] earned 500 total points
ID: 34154321
Mysql is able to handle a join between tables having millions of records, espacially if there are appropriate indexes.
0
 

Author Comment

by:andieje
ID: 34154337
this is what i thought but i've been warned by someone not to do what I was planning to do, but as you say they might have left the innodb_buffer_pool_size at 1M

I don't suppose youve ran mysql on the cloud have you? I was wondering how much memory they would let you assign to mysql. I'm not the 'cloud person' in my team. I'm not even the computer person :)
0
 
LVL 142

Expert Comment

by:Guy Hengel [angelIII / a3]
ID: 34154368
>I don't suppose youve ran mysql on the cloud have you?
no. only a couple of distinct instances, each for a very specific task each.
the largest installation has some 200 tables, each 4GB each table (1 table = 1 day of data, actually)
0
 

Author Comment

by:andieje
ID: 34154383
you've answered many of my questions before which i really appreciate so I know you know your stuff inside out!

I would really appreciate it if you could look at this question for me if you have any time
http://www.experts-exchange.com/Database/MySQL/Q_26619565.html

many thanks
0
 

Author Closing Comment

by:andieje
ID: 34201327
thanks
0

Featured Post

Microsoft Certification Exam 74-409

Veeam® is happy to provide the Microsoft community with a study guide prepared by MVP and MCT, Orin Thomas. This guide will take you through each of the exam objectives, helping you to prepare for and pass the examination.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

I use MySQL for many of my development projects in a Windows environment. To manage my databases (and perform queries) for years I used a tool called MySQL administrator.  This tool has since been replaced by MySQL Workbench. So I decided to m…
Password hashing is better than message digests or encryption, and you should be using it instead of message digests or encryption.  Find out why and how in this article, which supplements the original article on PHP Client Registration, Login, Logo…
This Micro Tutorial hows how you can integrate  Mac OSX to a Windows Active Directory Domain. Apple has made it easy to allow users to bind their macs to a windows domain with relative ease. The following video show how to bind OSX Mavericks to …
With the power of JIRA, there's an unlimited number of ways you can customize it, use it and benefit from it. With that in mind, there's bound to be things that I wasn't able to cover in this course. With this summary we'll look at some places to go…

863 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

23 Experts available now in Live!

Get 1:1 Help Now