Solved

how much data can mysql handle, really

Posted on 2010-11-17
11
721 Views
Last Modified: 2012-05-10
Hi

Is mysql capable of handling joins on really large tables?

I am having a problem working with large amounts of data. I tried to join a table with 1 million records to a table with 100,000 records and the join 'ate up' all the memory in the machine so that everything just died.

Mysql is an enterprise database but I have no idea if it can handle database joins on these types of scale.

Is there a way to help it out in these situations? I'm sure indexing will help but is there anyway of breaking a join down into blocks or steps

thanks
0
Comment
Question by:andieje
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 6
  • 5
11 Comments
 
LVL 143

Expert Comment

by:Guy Hengel [angelIII / a3]
ID: 34154230
table structure is?
indexes on the tables are?
query is?
explain plan of the query is?

configuration/tuning tips are here:
http://www.mysqlperformanceblog.com/2006/09/29/what-to-tune-in-mysql-server-after-installation/
0
 

Author Comment

by:andieje
ID: 34154291
I don't have a specific example, this is an exploratoty question

Simple one to many relationship
Lets say i have table A and table B. A contains a million records and table B contains a 100,000 records. There is a one to many relationship between table A and B. They have a common field which is the prinary key in table A and is a foreign key in table B. This field is indexed.

select * from table a, table b inner join on a.pk = b.fk

0
 
LVL 143

Expert Comment

by:Guy Hengel [angelIII / a3]
ID: 34154305
the correct syntax, in MySQL, would be either:
select * from table a join table b on a.pk = b.fk

Open in new window

or
select * from table a , table b where a.pk = b.fk

Open in new window


and when you say both pk and fk fields are indexes, the join should use the indexes to perform the join... eventually.
it does depend on the configuration of the memory assigned to mysql (see link above)
0
Windows Server 2016: All you need to know

Learn about Hyper-V features that increase functionality and usability of Microsoft Windows Server 2016. Also, throughout this eBook, you’ll find some basic PowerShell examples that will help you leverage the scripts in your environments!

 

Author Comment

by:andieje
ID: 34154308
I am asking this question because i have been told mysql won't handle the type of data I want it to handle because when someone else tried it, mysql simply 'died.' I'm hoping not to waste my time if I can avoid it

0
 
LVL 143

Expert Comment

by:Guy Hengel [angelIII / a3]
ID: 34154313
I have to presume that the "someone else" did not know about the configuration you have to do with MySQL to allow it to take more RAM than it does by default... which is usually much too low, by default ...
0
 

Author Comment

by:andieje
ID: 34154320
what do you mean the 'join should use the indexes to perform the join eventually'

I apologise for getting the syntax wrong. My question isn't about the syntax though its simple, should mysql be able to handle a simple query like that with that much data or would i really have to fine tune the configuration?
0
 
LVL 143

Accepted Solution

by:
Guy Hengel [angelIII / a3] earned 500 total points
ID: 34154321
Mysql is able to handle a join between tables having millions of records, espacially if there are appropriate indexes.
0
 

Author Comment

by:andieje
ID: 34154337
this is what i thought but i've been warned by someone not to do what I was planning to do, but as you say they might have left the innodb_buffer_pool_size at 1M

I don't suppose youve ran mysql on the cloud have you? I was wondering how much memory they would let you assign to mysql. I'm not the 'cloud person' in my team. I'm not even the computer person :)
0
 
LVL 143

Expert Comment

by:Guy Hengel [angelIII / a3]
ID: 34154368
>I don't suppose youve ran mysql on the cloud have you?
no. only a couple of distinct instances, each for a very specific task each.
the largest installation has some 200 tables, each 4GB each table (1 table = 1 day of data, actually)
0
 

Author Comment

by:andieje
ID: 34154383
you've answered many of my questions before which i really appreciate so I know you know your stuff inside out!

I would really appreciate it if you could look at this question for me if you have any time
http://www.experts-exchange.com/Database/MySQL/Q_26619565.html

many thanks
0
 

Author Closing Comment

by:andieje
ID: 34201327
thanks
0

Featured Post

What Is Transaction Monitoring and who needs it?

Synthetic Transaction Monitoring that you need for the day to day, which ensures your business website keeps running optimally, and that there is no downtime to impact your customer experience.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Foreword This is an old article.  Instead of using the MySQL extension that was used in the original code examples, please choose one of the currently supported database extensions instead.  More information is available here: MySQLi / PDO (http://…
When table data gets too large to manage or queries take too long to execute the solution is often to buy bigger hardware or assign more CPUs and memory resources to the machine to solve the problem. However, the best, cheapest and most effective so…
There's a multitude of different network monitoring solutions out there, and you're probably wondering what makes NetCrunch so special. It's completely agentless, but does let you create an agent, if you desire. It offers powerful scalability …
Monitoring a network: why having a policy is the best policy? Michael Kulchisky, MCSE, MCSA, MCP, VTSP, VSP, CCSP outlines the enormous benefits of having a policy-based approach when monitoring medium and large networks. Software utilized in this v…

734 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question