?
Solved

Speeding up MySQL update query

Posted on 2006-06-19
20
Medium Priority
?
297 Views
Last Modified: 2012-06-21
Ok, I've been working on this script for awhile and have one major bottleneck that I can't seem to get past.

As the table I am updating grows in size, this particular query takes longer and longer. Towards the end of processing it takes about a full second, which considering how many times it has to run, this slows down the whole process by several hours. There are only about 80k rows in the table at the end of processing.

The query basically looks like:

UPDATE table SET thumbnailname = 'IMAGENAME.JPG' WHERE id1=10 AND num=20902 ;

The names have been changed to protect the innocent, but that's the query.

I don't see why such a simple query should take so long, so I'm posting here.

Any help would be appreciated, If you need any more info, please let me know.
0
Comment
Question by:pcgentry
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 13
  • 7
20 Comments
 
LVL 30

Expert Comment

by:todd_farmer
ID: 16935765
Make sure you have an index on table(id1, num).
0
 

Author Comment

by:pcgentry
ID: 16935822
Already done. I should have mentioned that..

I've tried setting them up seperately and jointly as you said (id1, num)... which did actually give me a speed boost from where I was before that point, but still leaves me at the problem described above.
0
 
LVL 30

Expert Comment

by:todd_farmer
ID: 16935841
Can you show us the results of:

SHOW CREATE TABLE table;

EXPLAIN SELECT * FROM table WHERE id1=10 AND num=20902 ;
0
Prepare for your VMware VCP6-DCV exam.

Josh Coen and Jason Langer have prepared the latest edition of VCP study guide. Both authors have been working in the IT field for more than a decade, and both hold VMware certifications. This 163-page guide covers all 10 of the exam blueprint sections.

 

Author Comment

by:pcgentry
ID: 16935892
Sure, to heck with innocent ... here is the show create table as it is actually being used.

idx   |CREATE TABLE `idx` (
`mlsnum` varchar(10) NOT NULL COMMENT 'mls number, semi-unique id for property
not mls',
`propclass` varchar(30) default NULL,
`proptype` varchar(30) default NULL,
  `propclass` varchar(30) default NULL,
  `proptype` varchar(30) default NULL,
  `askprice` int(10) default NULL,
  `showaddress` tinyint(1) default '1',
  `address` varchar(255) default NULL,
  `address2` varchar(255) default NULL,
  `city` varchar(50) default NULL,
  `state` char(3) default NULL,
  `zipcode` varchar(10) default NULL,
  `sqft` int(7) default NULL,
  `sqft1` varchar(30) default NULL,
  `numbdrms` int(2) default NULL,
  `numfullbath` int(2) default NULL,
  `numhalfbath` int(2) default NULL,
  `garage` varchar(30) default NULL,
  `basement` varchar(30) default NULL,
  `acres` float(8,2) default NULL,
  `listingoffice` varchar(255) default NULL,
  `listingofficephone` varchar(30) default NULL,
  `subdivsn` varchar(30) default NULL,
  `county` varchar(30) default NULL,
  `school` varchar(30) default NULL,
  `level1` char(1) default NULL,
  `dimensions1` varchar(14) default NULL,
  `level2` char(1) default NULL,
  `dimensions2` varchar(14) default NULL,
  `level3` char(1) default NULL,
  `dimensions3` varchar(14) default NULL,
  `level4` char(1) default NULL,
  `dimensions4` varchar(14) default NULL,
  `level5` char(1) default NULL,
  `dimensions5` varchar(14) default NULL,
  `level6` char(1) default NULL,
  `dimensions6` varchar(14) default NULL,
  `level7` char(1) default NULL,
  `dimensions7` varchar(14) default NULL,
  `level8` char(1) default NULL,
  `dimensions8` varchar(14) default NULL,
  `level9` char(1) default NULL,
  `dimensions9` varchar(14) default NULL,
  `level10` char(1) default NULL,
  `dimensions10` varchar(14) default NULL,
  `level11` char(1) default NULL,
  `dimensions11` varchar(11) default NULL,
  `roomname` varchar(13) default NULL,
  `numunits` int(3) default NULL,
  `unit1a` char(3) default NULL,
  `unit1b` char(3) default NULL,
  `unit1c` varchar(6) default NULL,
  `unit2a` char(3) default NULL,
  `unit2b` char(3) default NULL,
  `unit2c` varchar(6) default NULL,
  `unit3a` char(3) default NULL,
  `unit3b` char(3) default NULL,
  `unit3c` varchar(6) default NULL,
  `unit4a` char(3) default NULL,
  `unit4b` char(3) default NULL,
  `unit4c` varchar(6) default NULL,
  `surfacewater` tinyint(1) default NULL,
  `features` varchar(255) default NULL,
  `remarks` varchar(255) default NULL,
  `thumbnail` tinyint(1) default NULL,
  `thumbnailname` varchar(255) default NULL,
  `photopath` varchar(255) default NULL,
  `agencyid_fk` int(10) default NULL,
  `taxes` varchar(6) default NULL,
  `AgentName` varchar(16) default NULL,
  `yearbuilt` int(4) default NULL,
  `mlsid` int(8) NOT NULL default '0',
  PRIMARY KEY  (`mlsnum`,`mlsid`),
  UNIQUE KEY `mlsid` (`mlsnum`,`mlsid`),
  KEY `test` (`mlsid`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1 |

Eplain coming in a few seconds....
0
 

Author Comment

by:pcgentry
ID: 16935920
+----+-------------+-------+------+--------------------+------+---------+-------+------+-------------+
| id | select_type | table | type | possible_keys      | key  | key_len | ref   | rows | Extra       |
+----+-------------+-------+------+--------------------+------+---------+-------+------+-------------+
|  1 | SIMPLE      | idx   | ref  | PRIMARY,mlsid,test | test | 4       | const | 1131 | Using where |
+----+-------------+-------+------+--------------------+------+---------+-------+------+-------------+
1 row in set (0.00 sec)
0
 
LVL 30

Expert Comment

by:todd_farmer
ID: 16935922
FYI, you don't need the UNIQUE KEY, since those columns are in the primary key (which is both unique and not null).
0
 
LVL 30

Expert Comment

by:todd_farmer
ID: 16935948
Try this:

EXPLAIN SELECT * FROM table IGNORE INDEX (test) WHERE id1=10 AND num=20902 ;
0
 

Author Comment

by:pcgentry
ID: 16935951
Ok, I noticed something there... that the select statement didn't even register time-wise. As I understand it, an update with the added overhead of a write... so I ran that particular update by itself and it ran in .06 seconds, which is an acceptable time.

So I am thinking that the problem must have somehting to do with having so many statements ran one after another?
0
 

Author Comment

by:pcgentry
ID: 16935964
RE: FYI, you don't need the UNIQUE KEY, since those columns are in the primary key (which is both unique and not null).

gotcha, what can I say... I've gotten desperate :)


mysql> EXPLAIN SELECT * FROM idx IGNORE INDEX (test) WHERE mlsid=10 AND mlsnum=20902 ;
+----+-------------+-------+------+---------------+------+---------+------+-------+-------------+
| id | select_type | table | type | possible_keys | key  | key_len | ref  | rows  | Extra       |
+----+-------------+-------+------+---------------+------+---------+------+-------+-------------+
|  1 | SIMPLE      | idx   | ALL  | PRIMARY,mlsid | NULL | NULL    | NULL | 81219 | Using where |
+----+-------------+-------+------+---------------+------+---------+------+-------+-------------+
1 row in set (0.02 sec)
0
 

Author Comment

by:pcgentry
ID: 16935997
To give a better idea of what is going on here, all the sql statements are dumped into text files (currently about 10) and run with a batch file.

The largest text file has about 31 k statements in it, about 1 out of 5 similiar to the statement we have been discussing.
0
 

Author Comment

by:pcgentry
ID: 16936069
All that put aside... I ran the script again and it is moving much faster.

I guess I overworked the problem.

Thanks so much for your help!
0
 

Author Comment

by:pcgentry
ID: 16936075
or not ... it hit one of the big files and slowed down again .. and I was so excited I almost posted points :)
0
 

Author Comment

by:pcgentry
ID: 16936096
I think the statements made before were for some of the update statements that apperently run much faster..

Take a look at this:


mysql> EXPLAIN SELECT * FROM idx IGNORE INDEX (test) WHERE mlsid=22 AND mlsnum=436306 ;
+----+-------------+-------+------+---------------+------+---------+------+-------+-------------+
| id | select_type | table | type | possible_keys | key  | key_len | ref  | rows  | Extra       |
+----+-------------+-------+------+---------------+------+---------+------+-------+-------------+
|  1 | SIMPLE      | idx   | ALL  | PRIMARY,mlsid | NULL | NULL    | NULL | 81219 | Using where |
+----+-------------+-------+------+---------------+------+---------+------+-------+-------------+
1 row in set (0.28 sec)
0
 

Author Comment

by:pcgentry
ID: 16936279
mysql> EXPLAIN SELECT * FROM idx IGNORE INDEX (test) WHERE mlsid=22 AND mlsnum=436306 ;
+----+-------------+-------+------+---------------+------+---------+------+-------+-------------+
| id | select_type | table | type | possible_keys | key  | key_len | ref  | rows  | Extra       |
+----+-------------+-------+------+---------------+------+---------+------+-------+-------------+
|  1 | SIMPLE      | idx   | ALL  | PRIMARY,mlsid | NULL | NULL    | NULL | 81219 | Using where |
+----+-------------+-------+------+---------------+------+---------+------+-------+-------------+
1 row in set (1.63 sec)

mysql> EXPLAIN SELECT * FROM idx IGNORE INDEX (test) WHERE mlsid=22 AND mlsnum=436306 ;
+----+-------------+-------+------+---------------+------+---------+------+-------+-------------+
| id | select_type | table | type | possible_keys | key  | key_len | ref  | rows  | Extra       |
+----+-------------+-------+------+---------------+------+---------+------+-------+-------------+
|  1 | SIMPLE      | idx   | ALL  | PRIMARY,mlsid | NULL | NULL    | NULL | 81219 | Using where |
+----+-------------+-------+------+---------------+------+---------+------+-------+-------------+
1 row in set (1.89 sec)


Things are still much too slow... still need some help here.
0
 
LVL 30

Expert Comment

by:todd_farmer
ID: 16936391
I think that perhaps the reason it is not using the index is that your mlsnum column is defined as VARCHAR, but you're passing in a number, which MySQL converts to a VARCHAR internally.  Try this:

EXPLAIN SELECT * FROM idx IGNORE INDEX (test) WHERE mlsid=22 AND mlsnum='436306' ;
0
 

Author Comment

by:pcgentry
ID: 16936526
mysql> EXPLAIN SELECT * FROM idx IGNORE INDEX (test) WHERE mlsid=22 AND mlsnum='436306' ;
+----+-------------+-------+------+----------------------+---------+---------+-------------+------+-------------+
| id | select_type | table | type | possible_keys        | key     | key_len | ref         | rows | Extra       |
+----+-------------+-------+------+----------------------+---------+---------+-------------+------+-------------+
|  1 | SIMPLE      | idx   | ref  | PRIMARY,mlsid,mlsnum | PRIMARY | 16      | const,const |    1 | Using where |
+----+-------------+-------+------+----------------------+---------+---------+-------------+------+-------------+
1 row in set (0.42 sec)



When repeated the it ranges in time between .17 and .95 sec

0
 
LVL 30

Accepted Solution

by:
todd_farmer earned 500 total points
ID: 16936543
OK, that sounds better.  So you would want to modify your update statement to be:

UPDATE table SET thumbnailname = 'IMAGENAME.JPG' WHERE id1=10 AND num='20902' ;
0
 

Author Comment

by:pcgentry
ID: 16936831
That did it... much better!

Anything else you can think of that might speed things up?
0
 
LVL 30

Expert Comment

by:todd_farmer
ID: 16936860
If the mlsnum really is a numeric value, I would redefine the column as such and revert to the original update statement.  Comparisons using numeric values as opposed to text will be faster, but probably not by a significant measure.
0
 

Author Comment

by:pcgentry
ID: 16936918
gotcha.. can't depend on it to be numeric in any case, alpha-numerics pop up every once in awhile.


The answers you already gave have taken care of the major blockage that caused me to post. So thnx a million:)

If you can come up with anything else that may help it more, I'll be glad to send over some more points.
0

Featured Post

Visualize your virtual and backup environments

Create well-organized and polished visualizations of your virtual and backup environments when planning VMware vSphere, Microsoft Hyper-V or Veeam deployments. It helps you to gain better visibility and valuable business insights.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

In this article, we’ll look at how to deploy ProxySQL.
In this article, I’ll talk about multi-threaded slave statistics printed in MySQL error log file.
In this video, Percona Solution Engineer Dimitri Vanoverbeke discusses why you want to use at least three nodes in a database cluster. To discuss how Percona Consulting can help with your design and architecture needs for your database and infras…
In this video, Percona Solutions Engineer Barrett Chambers discusses some of the basic syntax differences between MySQL and MongoDB. To learn more check out our webinar on MongoDB administration for MySQL DBA: https://www.percona.com/resources/we…
Suggested Courses

777 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question