Solved

Speeding up MySQL update query

Posted on 2006-06-19
20
285 Views
Last Modified: 2012-06-21
Ok, I've been working on this script for awhile and have one major bottleneck that I can't seem to get past.

As the table I am updating grows in size, this particular query takes longer and longer. Towards the end of processing it takes about a full second, which considering how many times it has to run, this slows down the whole process by several hours. There are only about 80k rows in the table at the end of processing.

The query basically looks like:

UPDATE table SET thumbnailname = 'IMAGENAME.JPG' WHERE id1=10 AND num=20902 ;

The names have been changed to protect the innocent, but that's the query.

I don't see why such a simple query should take so long, so I'm posting here.

Any help would be appreciated, If you need any more info, please let me know.
0
Comment
Question by:pcgentry
  • 13
  • 7
20 Comments
 
LVL 30

Expert Comment

by:todd_farmer
ID: 16935765
Make sure you have an index on table(id1, num).
0
 

Author Comment

by:pcgentry
ID: 16935822
Already done. I should have mentioned that..

I've tried setting them up seperately and jointly as you said (id1, num)... which did actually give me a speed boost from where I was before that point, but still leaves me at the problem described above.
0
 
LVL 30

Expert Comment

by:todd_farmer
ID: 16935841
Can you show us the results of:

SHOW CREATE TABLE table;

EXPLAIN SELECT * FROM table WHERE id1=10 AND num=20902 ;
0
 

Author Comment

by:pcgentry
ID: 16935892
Sure, to heck with innocent ... here is the show create table as it is actually being used.

idx   |CREATE TABLE `idx` (
`mlsnum` varchar(10) NOT NULL COMMENT 'mls number, semi-unique id for property
not mls',
`propclass` varchar(30) default NULL,
`proptype` varchar(30) default NULL,
  `propclass` varchar(30) default NULL,
  `proptype` varchar(30) default NULL,
  `askprice` int(10) default NULL,
  `showaddress` tinyint(1) default '1',
  `address` varchar(255) default NULL,
  `address2` varchar(255) default NULL,
  `city` varchar(50) default NULL,
  `state` char(3) default NULL,
  `zipcode` varchar(10) default NULL,
  `sqft` int(7) default NULL,
  `sqft1` varchar(30) default NULL,
  `numbdrms` int(2) default NULL,
  `numfullbath` int(2) default NULL,
  `numhalfbath` int(2) default NULL,
  `garage` varchar(30) default NULL,
  `basement` varchar(30) default NULL,
  `acres` float(8,2) default NULL,
  `listingoffice` varchar(255) default NULL,
  `listingofficephone` varchar(30) default NULL,
  `subdivsn` varchar(30) default NULL,
  `county` varchar(30) default NULL,
  `school` varchar(30) default NULL,
  `level1` char(1) default NULL,
  `dimensions1` varchar(14) default NULL,
  `level2` char(1) default NULL,
  `dimensions2` varchar(14) default NULL,
  `level3` char(1) default NULL,
  `dimensions3` varchar(14) default NULL,
  `level4` char(1) default NULL,
  `dimensions4` varchar(14) default NULL,
  `level5` char(1) default NULL,
  `dimensions5` varchar(14) default NULL,
  `level6` char(1) default NULL,
  `dimensions6` varchar(14) default NULL,
  `level7` char(1) default NULL,
  `dimensions7` varchar(14) default NULL,
  `level8` char(1) default NULL,
  `dimensions8` varchar(14) default NULL,
  `level9` char(1) default NULL,
  `dimensions9` varchar(14) default NULL,
  `level10` char(1) default NULL,
  `dimensions10` varchar(14) default NULL,
  `level11` char(1) default NULL,
  `dimensions11` varchar(11) default NULL,
  `roomname` varchar(13) default NULL,
  `numunits` int(3) default NULL,
  `unit1a` char(3) default NULL,
  `unit1b` char(3) default NULL,
  `unit1c` varchar(6) default NULL,
  `unit2a` char(3) default NULL,
  `unit2b` char(3) default NULL,
  `unit2c` varchar(6) default NULL,
  `unit3a` char(3) default NULL,
  `unit3b` char(3) default NULL,
  `unit3c` varchar(6) default NULL,
  `unit4a` char(3) default NULL,
  `unit4b` char(3) default NULL,
  `unit4c` varchar(6) default NULL,
  `surfacewater` tinyint(1) default NULL,
  `features` varchar(255) default NULL,
  `remarks` varchar(255) default NULL,
  `thumbnail` tinyint(1) default NULL,
  `thumbnailname` varchar(255) default NULL,
  `photopath` varchar(255) default NULL,
  `agencyid_fk` int(10) default NULL,
  `taxes` varchar(6) default NULL,
  `AgentName` varchar(16) default NULL,
  `yearbuilt` int(4) default NULL,
  `mlsid` int(8) NOT NULL default '0',
  PRIMARY KEY  (`mlsnum`,`mlsid`),
  UNIQUE KEY `mlsid` (`mlsnum`,`mlsid`),
  KEY `test` (`mlsid`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1 |

Eplain coming in a few seconds....
0
 

Author Comment

by:pcgentry
ID: 16935920
+----+-------------+-------+------+--------------------+------+---------+-------+------+-------------+
| id | select_type | table | type | possible_keys      | key  | key_len | ref   | rows | Extra       |
+----+-------------+-------+------+--------------------+------+---------+-------+------+-------------+
|  1 | SIMPLE      | idx   | ref  | PRIMARY,mlsid,test | test | 4       | const | 1131 | Using where |
+----+-------------+-------+------+--------------------+------+---------+-------+------+-------------+
1 row in set (0.00 sec)
0
 
LVL 30

Expert Comment

by:todd_farmer
ID: 16935922
FYI, you don't need the UNIQUE KEY, since those columns are in the primary key (which is both unique and not null).
0
 
LVL 30

Expert Comment

by:todd_farmer
ID: 16935948
Try this:

EXPLAIN SELECT * FROM table IGNORE INDEX (test) WHERE id1=10 AND num=20902 ;
0
 

Author Comment

by:pcgentry
ID: 16935951
Ok, I noticed something there... that the select statement didn't even register time-wise. As I understand it, an update with the added overhead of a write... so I ran that particular update by itself and it ran in .06 seconds, which is an acceptable time.

So I am thinking that the problem must have somehting to do with having so many statements ran one after another?
0
 

Author Comment

by:pcgentry
ID: 16935964
RE: FYI, you don't need the UNIQUE KEY, since those columns are in the primary key (which is both unique and not null).

gotcha, what can I say... I've gotten desperate :)


mysql> EXPLAIN SELECT * FROM idx IGNORE INDEX (test) WHERE mlsid=10 AND mlsnum=20902 ;
+----+-------------+-------+------+---------------+------+---------+------+-------+-------------+
| id | select_type | table | type | possible_keys | key  | key_len | ref  | rows  | Extra       |
+----+-------------+-------+------+---------------+------+---------+------+-------+-------------+
|  1 | SIMPLE      | idx   | ALL  | PRIMARY,mlsid | NULL | NULL    | NULL | 81219 | Using where |
+----+-------------+-------+------+---------------+------+---------+------+-------+-------------+
1 row in set (0.02 sec)
0
 

Author Comment

by:pcgentry
ID: 16935997
To give a better idea of what is going on here, all the sql statements are dumped into text files (currently about 10) and run with a batch file.

The largest text file has about 31 k statements in it, about 1 out of 5 similiar to the statement we have been discussing.
0
Control application downtime with dependency maps

Visualize the interdependencies between application components better with Applications Manager's automated application discovery and dependency mapping feature. Resolve performance issues faster by quickly isolating problematic components.

 

Author Comment

by:pcgentry
ID: 16936069
All that put aside... I ran the script again and it is moving much faster.

I guess I overworked the problem.

Thanks so much for your help!
0
 

Author Comment

by:pcgentry
ID: 16936075
or not ... it hit one of the big files and slowed down again .. and I was so excited I almost posted points :)
0
 

Author Comment

by:pcgentry
ID: 16936096
I think the statements made before were for some of the update statements that apperently run much faster..

Take a look at this:


mysql> EXPLAIN SELECT * FROM idx IGNORE INDEX (test) WHERE mlsid=22 AND mlsnum=436306 ;
+----+-------------+-------+------+---------------+------+---------+------+-------+-------------+
| id | select_type | table | type | possible_keys | key  | key_len | ref  | rows  | Extra       |
+----+-------------+-------+------+---------------+------+---------+------+-------+-------------+
|  1 | SIMPLE      | idx   | ALL  | PRIMARY,mlsid | NULL | NULL    | NULL | 81219 | Using where |
+----+-------------+-------+------+---------------+------+---------+------+-------+-------------+
1 row in set (0.28 sec)
0
 

Author Comment

by:pcgentry
ID: 16936279
mysql> EXPLAIN SELECT * FROM idx IGNORE INDEX (test) WHERE mlsid=22 AND mlsnum=436306 ;
+----+-------------+-------+------+---------------+------+---------+------+-------+-------------+
| id | select_type | table | type | possible_keys | key  | key_len | ref  | rows  | Extra       |
+----+-------------+-------+------+---------------+------+---------+------+-------+-------------+
|  1 | SIMPLE      | idx   | ALL  | PRIMARY,mlsid | NULL | NULL    | NULL | 81219 | Using where |
+----+-------------+-------+------+---------------+------+---------+------+-------+-------------+
1 row in set (1.63 sec)

mysql> EXPLAIN SELECT * FROM idx IGNORE INDEX (test) WHERE mlsid=22 AND mlsnum=436306 ;
+----+-------------+-------+------+---------------+------+---------+------+-------+-------------+
| id | select_type | table | type | possible_keys | key  | key_len | ref  | rows  | Extra       |
+----+-------------+-------+------+---------------+------+---------+------+-------+-------------+
|  1 | SIMPLE      | idx   | ALL  | PRIMARY,mlsid | NULL | NULL    | NULL | 81219 | Using where |
+----+-------------+-------+------+---------------+------+---------+------+-------+-------------+
1 row in set (1.89 sec)


Things are still much too slow... still need some help here.
0
 
LVL 30

Expert Comment

by:todd_farmer
ID: 16936391
I think that perhaps the reason it is not using the index is that your mlsnum column is defined as VARCHAR, but you're passing in a number, which MySQL converts to a VARCHAR internally.  Try this:

EXPLAIN SELECT * FROM idx IGNORE INDEX (test) WHERE mlsid=22 AND mlsnum='436306' ;
0
 

Author Comment

by:pcgentry
ID: 16936526
mysql> EXPLAIN SELECT * FROM idx IGNORE INDEX (test) WHERE mlsid=22 AND mlsnum='436306' ;
+----+-------------+-------+------+----------------------+---------+---------+-------------+------+-------------+
| id | select_type | table | type | possible_keys        | key     | key_len | ref         | rows | Extra       |
+----+-------------+-------+------+----------------------+---------+---------+-------------+------+-------------+
|  1 | SIMPLE      | idx   | ref  | PRIMARY,mlsid,mlsnum | PRIMARY | 16      | const,const |    1 | Using where |
+----+-------------+-------+------+----------------------+---------+---------+-------------+------+-------------+
1 row in set (0.42 sec)



When repeated the it ranges in time between .17 and .95 sec

0
 
LVL 30

Accepted Solution

by:
todd_farmer earned 125 total points
ID: 16936543
OK, that sounds better.  So you would want to modify your update statement to be:

UPDATE table SET thumbnailname = 'IMAGENAME.JPG' WHERE id1=10 AND num='20902' ;
0
 

Author Comment

by:pcgentry
ID: 16936831
That did it... much better!

Anything else you can think of that might speed things up?
0
 
LVL 30

Expert Comment

by:todd_farmer
ID: 16936860
If the mlsnum really is a numeric value, I would redefine the column as such and revert to the original update statement.  Comparisons using numeric values as opposed to text will be faster, but probably not by a significant measure.
0
 

Author Comment

by:pcgentry
ID: 16936918
gotcha.. can't depend on it to be numeric in any case, alpha-numerics pop up every once in awhile.


The answers you already gave have taken care of the major blockage that caused me to post. So thnx a million:)

If you can come up with anything else that may help it more, I'll be glad to send over some more points.
0

Featured Post

Backup Your Microsoft Windows Server®

Backup all your Microsoft Windows Server – on-premises, in remote locations, in private and hybrid clouds. Your entire Windows Server will be backed up in one easy step with patented, block-level disk imaging. We achieve RTOs (recovery time objectives) as low as 15 seconds.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
MySql hide the stored procedures 2 39
Sql query on a varchar that is numeric. 8 27
MySqlDump not dumping triggers 1 18
change database name 2 4
I have been using r1soft Continuous Data Protection (http://www.r1soft.com/linux-cdp/) for many years now with the mySQL Addon and wanted to share a trick I have used several times. For those of us that don't have the luxury of using all transact…
Popularity Can Be Measured Sometimes we deal with questions of popularity, and we need a way to collect opinions from our clients.  This article shows a simple teaching example of how we might elect a favorite color by letting our clients vote for …
This Micro Tutorial demonstrates using Microsoft Excel pivot tables, how to reverse engineer competitors' marketing strategies through backlinks.
Internet Business Fax to Email Made Easy - With  eFax Corporate (http://www.enterprise.efax.com), you'll receive a dedicated online fax number, which is used the same way as a typical analog fax number. You'll receive secure faxes in your email, f…

895 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

11 Experts available now in Live!

Get 1:1 Help Now