Solved

Speeding up MySQL update query

Posted on 2006-06-19
20
288 Views
Last Modified: 2012-06-21
Ok, I've been working on this script for awhile and have one major bottleneck that I can't seem to get past.

As the table I am updating grows in size, this particular query takes longer and longer. Towards the end of processing it takes about a full second, which considering how many times it has to run, this slows down the whole process by several hours. There are only about 80k rows in the table at the end of processing.

The query basically looks like:

UPDATE table SET thumbnailname = 'IMAGENAME.JPG' WHERE id1=10 AND num=20902 ;

The names have been changed to protect the innocent, but that's the query.

I don't see why such a simple query should take so long, so I'm posting here.

Any help would be appreciated, If you need any more info, please let me know.
0
Comment
Question by:pcgentry
  • 13
  • 7
20 Comments
 
LVL 30

Expert Comment

by:todd_farmer
ID: 16935765
Make sure you have an index on table(id1, num).
0
 

Author Comment

by:pcgentry
ID: 16935822
Already done. I should have mentioned that..

I've tried setting them up seperately and jointly as you said (id1, num)... which did actually give me a speed boost from where I was before that point, but still leaves me at the problem described above.
0
 
LVL 30

Expert Comment

by:todd_farmer
ID: 16935841
Can you show us the results of:

SHOW CREATE TABLE table;

EXPLAIN SELECT * FROM table WHERE id1=10 AND num=20902 ;
0
PRTG Network Monitor: Intuitive Network Monitoring

Network Monitoring is essential to ensure that computer systems and network devices are running. Use PRTG to monitor LANs, servers, websites, applications and devices, bandwidth, virtual environments, remote systems, IoT, and many more. PRTG is easy to set up & use.

 

Author Comment

by:pcgentry
ID: 16935892
Sure, to heck with innocent ... here is the show create table as it is actually being used.

idx   |CREATE TABLE `idx` (
`mlsnum` varchar(10) NOT NULL COMMENT 'mls number, semi-unique id for property
not mls',
`propclass` varchar(30) default NULL,
`proptype` varchar(30) default NULL,
  `propclass` varchar(30) default NULL,
  `proptype` varchar(30) default NULL,
  `askprice` int(10) default NULL,
  `showaddress` tinyint(1) default '1',
  `address` varchar(255) default NULL,
  `address2` varchar(255) default NULL,
  `city` varchar(50) default NULL,
  `state` char(3) default NULL,
  `zipcode` varchar(10) default NULL,
  `sqft` int(7) default NULL,
  `sqft1` varchar(30) default NULL,
  `numbdrms` int(2) default NULL,
  `numfullbath` int(2) default NULL,
  `numhalfbath` int(2) default NULL,
  `garage` varchar(30) default NULL,
  `basement` varchar(30) default NULL,
  `acres` float(8,2) default NULL,
  `listingoffice` varchar(255) default NULL,
  `listingofficephone` varchar(30) default NULL,
  `subdivsn` varchar(30) default NULL,
  `county` varchar(30) default NULL,
  `school` varchar(30) default NULL,
  `level1` char(1) default NULL,
  `dimensions1` varchar(14) default NULL,
  `level2` char(1) default NULL,
  `dimensions2` varchar(14) default NULL,
  `level3` char(1) default NULL,
  `dimensions3` varchar(14) default NULL,
  `level4` char(1) default NULL,
  `dimensions4` varchar(14) default NULL,
  `level5` char(1) default NULL,
  `dimensions5` varchar(14) default NULL,
  `level6` char(1) default NULL,
  `dimensions6` varchar(14) default NULL,
  `level7` char(1) default NULL,
  `dimensions7` varchar(14) default NULL,
  `level8` char(1) default NULL,
  `dimensions8` varchar(14) default NULL,
  `level9` char(1) default NULL,
  `dimensions9` varchar(14) default NULL,
  `level10` char(1) default NULL,
  `dimensions10` varchar(14) default NULL,
  `level11` char(1) default NULL,
  `dimensions11` varchar(11) default NULL,
  `roomname` varchar(13) default NULL,
  `numunits` int(3) default NULL,
  `unit1a` char(3) default NULL,
  `unit1b` char(3) default NULL,
  `unit1c` varchar(6) default NULL,
  `unit2a` char(3) default NULL,
  `unit2b` char(3) default NULL,
  `unit2c` varchar(6) default NULL,
  `unit3a` char(3) default NULL,
  `unit3b` char(3) default NULL,
  `unit3c` varchar(6) default NULL,
  `unit4a` char(3) default NULL,
  `unit4b` char(3) default NULL,
  `unit4c` varchar(6) default NULL,
  `surfacewater` tinyint(1) default NULL,
  `features` varchar(255) default NULL,
  `remarks` varchar(255) default NULL,
  `thumbnail` tinyint(1) default NULL,
  `thumbnailname` varchar(255) default NULL,
  `photopath` varchar(255) default NULL,
  `agencyid_fk` int(10) default NULL,
  `taxes` varchar(6) default NULL,
  `AgentName` varchar(16) default NULL,
  `yearbuilt` int(4) default NULL,
  `mlsid` int(8) NOT NULL default '0',
  PRIMARY KEY  (`mlsnum`,`mlsid`),
  UNIQUE KEY `mlsid` (`mlsnum`,`mlsid`),
  KEY `test` (`mlsid`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1 |

Eplain coming in a few seconds....
0
 

Author Comment

by:pcgentry
ID: 16935920
+----+-------------+-------+------+--------------------+------+---------+-------+------+-------------+
| id | select_type | table | type | possible_keys      | key  | key_len | ref   | rows | Extra       |
+----+-------------+-------+------+--------------------+------+---------+-------+------+-------------+
|  1 | SIMPLE      | idx   | ref  | PRIMARY,mlsid,test | test | 4       | const | 1131 | Using where |
+----+-------------+-------+------+--------------------+------+---------+-------+------+-------------+
1 row in set (0.00 sec)
0
 
LVL 30

Expert Comment

by:todd_farmer
ID: 16935922
FYI, you don't need the UNIQUE KEY, since those columns are in the primary key (which is both unique and not null).
0
 
LVL 30

Expert Comment

by:todd_farmer
ID: 16935948
Try this:

EXPLAIN SELECT * FROM table IGNORE INDEX (test) WHERE id1=10 AND num=20902 ;
0
 

Author Comment

by:pcgentry
ID: 16935951
Ok, I noticed something there... that the select statement didn't even register time-wise. As I understand it, an update with the added overhead of a write... so I ran that particular update by itself and it ran in .06 seconds, which is an acceptable time.

So I am thinking that the problem must have somehting to do with having so many statements ran one after another?
0
 

Author Comment

by:pcgentry
ID: 16935964
RE: FYI, you don't need the UNIQUE KEY, since those columns are in the primary key (which is both unique and not null).

gotcha, what can I say... I've gotten desperate :)


mysql> EXPLAIN SELECT * FROM idx IGNORE INDEX (test) WHERE mlsid=10 AND mlsnum=20902 ;
+----+-------------+-------+------+---------------+------+---------+------+-------+-------------+
| id | select_type | table | type | possible_keys | key  | key_len | ref  | rows  | Extra       |
+----+-------------+-------+------+---------------+------+---------+------+-------+-------------+
|  1 | SIMPLE      | idx   | ALL  | PRIMARY,mlsid | NULL | NULL    | NULL | 81219 | Using where |
+----+-------------+-------+------+---------------+------+---------+------+-------+-------------+
1 row in set (0.02 sec)
0
 

Author Comment

by:pcgentry
ID: 16935997
To give a better idea of what is going on here, all the sql statements are dumped into text files (currently about 10) and run with a batch file.

The largest text file has about 31 k statements in it, about 1 out of 5 similiar to the statement we have been discussing.
0
 

Author Comment

by:pcgentry
ID: 16936069
All that put aside... I ran the script again and it is moving much faster.

I guess I overworked the problem.

Thanks so much for your help!
0
 

Author Comment

by:pcgentry
ID: 16936075
or not ... it hit one of the big files and slowed down again .. and I was so excited I almost posted points :)
0
 

Author Comment

by:pcgentry
ID: 16936096
I think the statements made before were for some of the update statements that apperently run much faster..

Take a look at this:


mysql> EXPLAIN SELECT * FROM idx IGNORE INDEX (test) WHERE mlsid=22 AND mlsnum=436306 ;
+----+-------------+-------+------+---------------+------+---------+------+-------+-------------+
| id | select_type | table | type | possible_keys | key  | key_len | ref  | rows  | Extra       |
+----+-------------+-------+------+---------------+------+---------+------+-------+-------------+
|  1 | SIMPLE      | idx   | ALL  | PRIMARY,mlsid | NULL | NULL    | NULL | 81219 | Using where |
+----+-------------+-------+------+---------------+------+---------+------+-------+-------------+
1 row in set (0.28 sec)
0
 

Author Comment

by:pcgentry
ID: 16936279
mysql> EXPLAIN SELECT * FROM idx IGNORE INDEX (test) WHERE mlsid=22 AND mlsnum=436306 ;
+----+-------------+-------+------+---------------+------+---------+------+-------+-------------+
| id | select_type | table | type | possible_keys | key  | key_len | ref  | rows  | Extra       |
+----+-------------+-------+------+---------------+------+---------+------+-------+-------------+
|  1 | SIMPLE      | idx   | ALL  | PRIMARY,mlsid | NULL | NULL    | NULL | 81219 | Using where |
+----+-------------+-------+------+---------------+------+---------+------+-------+-------------+
1 row in set (1.63 sec)

mysql> EXPLAIN SELECT * FROM idx IGNORE INDEX (test) WHERE mlsid=22 AND mlsnum=436306 ;
+----+-------------+-------+------+---------------+------+---------+------+-------+-------------+
| id | select_type | table | type | possible_keys | key  | key_len | ref  | rows  | Extra       |
+----+-------------+-------+------+---------------+------+---------+------+-------+-------------+
|  1 | SIMPLE      | idx   | ALL  | PRIMARY,mlsid | NULL | NULL    | NULL | 81219 | Using where |
+----+-------------+-------+------+---------------+------+---------+------+-------+-------------+
1 row in set (1.89 sec)


Things are still much too slow... still need some help here.
0
 
LVL 30

Expert Comment

by:todd_farmer
ID: 16936391
I think that perhaps the reason it is not using the index is that your mlsnum column is defined as VARCHAR, but you're passing in a number, which MySQL converts to a VARCHAR internally.  Try this:

EXPLAIN SELECT * FROM idx IGNORE INDEX (test) WHERE mlsid=22 AND mlsnum='436306' ;
0
 

Author Comment

by:pcgentry
ID: 16936526
mysql> EXPLAIN SELECT * FROM idx IGNORE INDEX (test) WHERE mlsid=22 AND mlsnum='436306' ;
+----+-------------+-------+------+----------------------+---------+---------+-------------+------+-------------+
| id | select_type | table | type | possible_keys        | key     | key_len | ref         | rows | Extra       |
+----+-------------+-------+------+----------------------+---------+---------+-------------+------+-------------+
|  1 | SIMPLE      | idx   | ref  | PRIMARY,mlsid,mlsnum | PRIMARY | 16      | const,const |    1 | Using where |
+----+-------------+-------+------+----------------------+---------+---------+-------------+------+-------------+
1 row in set (0.42 sec)



When repeated the it ranges in time between .17 and .95 sec

0
 
LVL 30

Accepted Solution

by:
todd_farmer earned 125 total points
ID: 16936543
OK, that sounds better.  So you would want to modify your update statement to be:

UPDATE table SET thumbnailname = 'IMAGENAME.JPG' WHERE id1=10 AND num='20902' ;
0
 

Author Comment

by:pcgentry
ID: 16936831
That did it... much better!

Anything else you can think of that might speed things up?
0
 
LVL 30

Expert Comment

by:todd_farmer
ID: 16936860
If the mlsnum really is a numeric value, I would redefine the column as such and revert to the original update statement.  Comparisons using numeric values as opposed to text will be faster, but probably not by a significant measure.
0
 

Author Comment

by:pcgentry
ID: 16936918
gotcha.. can't depend on it to be numeric in any case, alpha-numerics pop up every once in awhile.


The answers you already gave have taken care of the major blockage that caused me to post. So thnx a million:)

If you can come up with anything else that may help it more, I'll be glad to send over some more points.
0

Featured Post

Gigs: Get Your Project Delivered by an Expert

Select from freelancers specializing in everything from database administration to programming, who have proven themselves as experts in their field. Hire the best, collaborate easily, pay securely and get projects done right.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Popularity Can Be Measured Sometimes we deal with questions of popularity, and we need a way to collect opinions from our clients.  This article shows a simple teaching example of how we might elect a favorite color by letting our clients vote for …
Both Easy and Powerful How easy is PHP? http://lmgtfy.com?q=how+easy+is+php  Very easy.  It has been described as "a programming language even my grandmother can use." How powerful is PHP?  http://en.wikipedia.org/wiki/PHP  Very powerful.  But a…
This Micro Tutorial will teach you how to censor certain areas of your screen. The example in this video will show a little boy's face being blurred. This will be demonstrated using Adobe Premiere Pro CS6.
This video shows how to quickly and easily add an email signature for all users on Exchange 2016. The resulting signature is applied on a server level by Exchange Online. The email signature template has been downloaded from: www.mail-signatures…

776 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question