Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people, just like you, are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
Solved

how much data can mysql handle, really

Posted on 2010-11-17
11
716 Views
Last Modified: 2012-05-10
Hi

Is mysql capable of handling joins on really large tables?

I am having a problem working with large amounts of data. I tried to join a table with 1 million records to a table with 100,000 records and the join 'ate up' all the memory in the machine so that everything just died.

Mysql is an enterprise database but I have no idea if it can handle database joins on these types of scale.

Is there a way to help it out in these situations? I'm sure indexing will help but is there anyway of breaking a join down into blocks or steps

thanks
0
Comment
Question by:andieje
  • 6
  • 5
11 Comments
 
LVL 143

Expert Comment

by:Guy Hengel [angelIII / a3]
ID: 34154230
table structure is?
indexes on the tables are?
query is?
explain plan of the query is?

configuration/tuning tips are here:
http://www.mysqlperformanceblog.com/2006/09/29/what-to-tune-in-mysql-server-after-installation/
0
 

Author Comment

by:andieje
ID: 34154291
I don't have a specific example, this is an exploratoty question

Simple one to many relationship
Lets say i have table A and table B. A contains a million records and table B contains a 100,000 records. There is a one to many relationship between table A and B. They have a common field which is the prinary key in table A and is a foreign key in table B. This field is indexed.

select * from table a, table b inner join on a.pk = b.fk

0
 
LVL 143

Expert Comment

by:Guy Hengel [angelIII / a3]
ID: 34154305
the correct syntax, in MySQL, would be either:
select * from table a join table b on a.pk = b.fk

Open in new window

or
select * from table a , table b where a.pk = b.fk

Open in new window


and when you say both pk and fk fields are indexes, the join should use the indexes to perform the join... eventually.
it does depend on the configuration of the memory assigned to mysql (see link above)
0
Windows Server 2016: All you need to know

Learn about Hyper-V features that increase functionality and usability of Microsoft Windows Server 2016. Also, throughout this eBook, you’ll find some basic PowerShell examples that will help you leverage the scripts in your environments!

 

Author Comment

by:andieje
ID: 34154308
I am asking this question because i have been told mysql won't handle the type of data I want it to handle because when someone else tried it, mysql simply 'died.' I'm hoping not to waste my time if I can avoid it

0
 
LVL 143

Expert Comment

by:Guy Hengel [angelIII / a3]
ID: 34154313
I have to presume that the "someone else" did not know about the configuration you have to do with MySQL to allow it to take more RAM than it does by default... which is usually much too low, by default ...
0
 

Author Comment

by:andieje
ID: 34154320
what do you mean the 'join should use the indexes to perform the join eventually'

I apologise for getting the syntax wrong. My question isn't about the syntax though its simple, should mysql be able to handle a simple query like that with that much data or would i really have to fine tune the configuration?
0
 
LVL 143

Accepted Solution

by:
Guy Hengel [angelIII / a3] earned 500 total points
ID: 34154321
Mysql is able to handle a join between tables having millions of records, espacially if there are appropriate indexes.
0
 

Author Comment

by:andieje
ID: 34154337
this is what i thought but i've been warned by someone not to do what I was planning to do, but as you say they might have left the innodb_buffer_pool_size at 1M

I don't suppose youve ran mysql on the cloud have you? I was wondering how much memory they would let you assign to mysql. I'm not the 'cloud person' in my team. I'm not even the computer person :)
0
 
LVL 143

Expert Comment

by:Guy Hengel [angelIII / a3]
ID: 34154368
>I don't suppose youve ran mysql on the cloud have you?
no. only a couple of distinct instances, each for a very specific task each.
the largest installation has some 200 tables, each 4GB each table (1 table = 1 day of data, actually)
0
 

Author Comment

by:andieje
ID: 34154383
you've answered many of my questions before which i really appreciate so I know you know your stuff inside out!

I would really appreciate it if you could look at this question for me if you have any time
http://www.experts-exchange.com/Database/MySQL/Q_26619565.html

many thanks
0
 

Author Closing Comment

by:andieje
ID: 34201327
thanks
0

Featured Post

PRTG Network Monitor: Intuitive Network Monitoring

Network Monitoring is essential to ensure that computer systems and network devices are running. Use PRTG to monitor LANs, servers, websites, applications and devices, bandwidth, virtual environments, remote systems, IoT, and many more. PRTG is easy to set up & use.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
PHP loop not working 4 71
count download link and run update query 9 82
PHP: Insert Data into MySQL 5 56
MySQL Memory Keeps Increasing 4 47
This guide whil teach how to setup live replication (database mirroring) on 2 servers for backup or other purposes. In our example situation we have this network schema (see atachment). We need to replicate EVERY executed SQL query on server 1 to…
Does the idea of dealing with bits scare or confuse you? Does it seem like a waste of time in an age where we all have terabytes of storage? If so, you're missing out on one of the core tools in every professional programmer's toolbox. Learn how to …
This video shows how to quickly and easily add an email signature for all users on Exchange 2016. The resulting signature is applied on a server level by Exchange Online. The email signature template has been downloaded from: www.mail-signatures…
A short tutorial showing how to set up an email signature in Outlook on the Web (previously known as OWA). For free email signatures designs, visit https://www.mail-signatures.com/articles/signature-templates/?sts=6651 If you want to manage em…

856 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question