how much data can mysql handle, really

Hi

Is mysql capable of handling joins on really large tables?

I am having a problem working with large amounts of data. I tried to join a table with 1 million records to a table with 100,000 records and the join 'ate up' all the memory in the machine so that everything just died.

Mysql is an enterprise database but I have no idea if it can handle database joins on these types of scale.

Is there a way to help it out in these situations? I'm sure indexing will help but is there anyway of breaking a join down into blocks or steps

thanks
andiejeAsked:
Who is Participating?
 
Guy Hengel [angelIII / a3]Connect With a Mentor Billing EngineerCommented:
Mysql is able to handle a join between tables having millions of records, espacially if there are appropriate indexes.
0
 
Guy Hengel [angelIII / a3]Billing EngineerCommented:
table structure is?
indexes on the tables are?
query is?
explain plan of the query is?

configuration/tuning tips are here:
http://www.mysqlperformanceblog.com/2006/09/29/what-to-tune-in-mysql-server-after-installation/
0
 
andiejeAuthor Commented:
I don't have a specific example, this is an exploratoty question

Simple one to many relationship
Lets say i have table A and table B. A contains a million records and table B contains a 100,000 records. There is a one to many relationship between table A and B. They have a common field which is the prinary key in table A and is a foreign key in table B. This field is indexed.

select * from table a, table b inner join on a.pk = b.fk

0
Free Tool: Path Explorer

An intuitive utility to help find the CSS path to UI elements on a webpage. These paths are used frequently in a variety of front-end development and QA automation tasks.

One of a set of tools we're offering as a way of saying thank you for being a part of the community.

 
Guy Hengel [angelIII / a3]Billing EngineerCommented:
the correct syntax, in MySQL, would be either:
select * from table a join table b on a.pk = b.fk

Open in new window

or
select * from table a , table b where a.pk = b.fk

Open in new window


and when you say both pk and fk fields are indexes, the join should use the indexes to perform the join... eventually.
it does depend on the configuration of the memory assigned to mysql (see link above)
0
 
andiejeAuthor Commented:
I am asking this question because i have been told mysql won't handle the type of data I want it to handle because when someone else tried it, mysql simply 'died.' I'm hoping not to waste my time if I can avoid it

0
 
Guy Hengel [angelIII / a3]Billing EngineerCommented:
I have to presume that the "someone else" did not know about the configuration you have to do with MySQL to allow it to take more RAM than it does by default... which is usually much too low, by default ...
0
 
andiejeAuthor Commented:
what do you mean the 'join should use the indexes to perform the join eventually'

I apologise for getting the syntax wrong. My question isn't about the syntax though its simple, should mysql be able to handle a simple query like that with that much data or would i really have to fine tune the configuration?
0
 
andiejeAuthor Commented:
this is what i thought but i've been warned by someone not to do what I was planning to do, but as you say they might have left the innodb_buffer_pool_size at 1M

I don't suppose youve ran mysql on the cloud have you? I was wondering how much memory they would let you assign to mysql. I'm not the 'cloud person' in my team. I'm not even the computer person :)
0
 
Guy Hengel [angelIII / a3]Billing EngineerCommented:
>I don't suppose youve ran mysql on the cloud have you?
no. only a couple of distinct instances, each for a very specific task each.
the largest installation has some 200 tables, each 4GB each table (1 table = 1 day of data, actually)
0
 
andiejeAuthor Commented:
you've answered many of my questions before which i really appreciate so I know you know your stuff inside out!

I would really appreciate it if you could look at this question for me if you have any time
http://www.experts-exchange.com/Database/MySQL/Q_26619565.html

many thanks
0
 
andiejeAuthor Commented:
thanks
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.