Solved

Speeding up MySQL update query

Posted on 2006-06-19
20
296 Views
Last Modified: 2012-06-21
Ok, I've been working on this script for awhile and have one major bottleneck that I can't seem to get past.

As the table I am updating grows in size, this particular query takes longer and longer. Towards the end of processing it takes about a full second, which considering how many times it has to run, this slows down the whole process by several hours. There are only about 80k rows in the table at the end of processing.

The query basically looks like:

UPDATE table SET thumbnailname = 'IMAGENAME.JPG' WHERE id1=10 AND num=20902 ;

The names have been changed to protect the innocent, but that's the query.

I don't see why such a simple query should take so long, so I'm posting here.

Any help would be appreciated, If you need any more info, please let me know.
0
Comment
Question by:pcgentry
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 13
  • 7
20 Comments
 
LVL 30

Expert Comment

by:todd_farmer
ID: 16935765
Make sure you have an index on table(id1, num).
0
 

Author Comment

by:pcgentry
ID: 16935822
Already done. I should have mentioned that..

I've tried setting them up seperately and jointly as you said (id1, num)... which did actually give me a speed boost from where I was before that point, but still leaves me at the problem described above.
0
 
LVL 30

Expert Comment

by:todd_farmer
ID: 16935841
Can you show us the results of:

SHOW CREATE TABLE table;

EXPLAIN SELECT * FROM table WHERE id1=10 AND num=20902 ;
0
Transaction Monitoring Vs. Real User Monitoring

Synthetic Transaction Monitoring Vs. Real User Monitoring: When To Use Each Approach? In this article, we will discuss two major monitoring approaches: Synthetic Transaction and Real User Monitoring.

 

Author Comment

by:pcgentry
ID: 16935892
Sure, to heck with innocent ... here is the show create table as it is actually being used.

idx   |CREATE TABLE `idx` (
`mlsnum` varchar(10) NOT NULL COMMENT 'mls number, semi-unique id for property
not mls',
`propclass` varchar(30) default NULL,
`proptype` varchar(30) default NULL,
  `propclass` varchar(30) default NULL,
  `proptype` varchar(30) default NULL,
  `askprice` int(10) default NULL,
  `showaddress` tinyint(1) default '1',
  `address` varchar(255) default NULL,
  `address2` varchar(255) default NULL,
  `city` varchar(50) default NULL,
  `state` char(3) default NULL,
  `zipcode` varchar(10) default NULL,
  `sqft` int(7) default NULL,
  `sqft1` varchar(30) default NULL,
  `numbdrms` int(2) default NULL,
  `numfullbath` int(2) default NULL,
  `numhalfbath` int(2) default NULL,
  `garage` varchar(30) default NULL,
  `basement` varchar(30) default NULL,
  `acres` float(8,2) default NULL,
  `listingoffice` varchar(255) default NULL,
  `listingofficephone` varchar(30) default NULL,
  `subdivsn` varchar(30) default NULL,
  `county` varchar(30) default NULL,
  `school` varchar(30) default NULL,
  `level1` char(1) default NULL,
  `dimensions1` varchar(14) default NULL,
  `level2` char(1) default NULL,
  `dimensions2` varchar(14) default NULL,
  `level3` char(1) default NULL,
  `dimensions3` varchar(14) default NULL,
  `level4` char(1) default NULL,
  `dimensions4` varchar(14) default NULL,
  `level5` char(1) default NULL,
  `dimensions5` varchar(14) default NULL,
  `level6` char(1) default NULL,
  `dimensions6` varchar(14) default NULL,
  `level7` char(1) default NULL,
  `dimensions7` varchar(14) default NULL,
  `level8` char(1) default NULL,
  `dimensions8` varchar(14) default NULL,
  `level9` char(1) default NULL,
  `dimensions9` varchar(14) default NULL,
  `level10` char(1) default NULL,
  `dimensions10` varchar(14) default NULL,
  `level11` char(1) default NULL,
  `dimensions11` varchar(11) default NULL,
  `roomname` varchar(13) default NULL,
  `numunits` int(3) default NULL,
  `unit1a` char(3) default NULL,
  `unit1b` char(3) default NULL,
  `unit1c` varchar(6) default NULL,
  `unit2a` char(3) default NULL,
  `unit2b` char(3) default NULL,
  `unit2c` varchar(6) default NULL,
  `unit3a` char(3) default NULL,
  `unit3b` char(3) default NULL,
  `unit3c` varchar(6) default NULL,
  `unit4a` char(3) default NULL,
  `unit4b` char(3) default NULL,
  `unit4c` varchar(6) default NULL,
  `surfacewater` tinyint(1) default NULL,
  `features` varchar(255) default NULL,
  `remarks` varchar(255) default NULL,
  `thumbnail` tinyint(1) default NULL,
  `thumbnailname` varchar(255) default NULL,
  `photopath` varchar(255) default NULL,
  `agencyid_fk` int(10) default NULL,
  `taxes` varchar(6) default NULL,
  `AgentName` varchar(16) default NULL,
  `yearbuilt` int(4) default NULL,
  `mlsid` int(8) NOT NULL default '0',
  PRIMARY KEY  (`mlsnum`,`mlsid`),
  UNIQUE KEY `mlsid` (`mlsnum`,`mlsid`),
  KEY `test` (`mlsid`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1 |

Eplain coming in a few seconds....
0
 

Author Comment

by:pcgentry
ID: 16935920
+----+-------------+-------+------+--------------------+------+---------+-------+------+-------------+
| id | select_type | table | type | possible_keys      | key  | key_len | ref   | rows | Extra       |
+----+-------------+-------+------+--------------------+------+---------+-------+------+-------------+
|  1 | SIMPLE      | idx   | ref  | PRIMARY,mlsid,test | test | 4       | const | 1131 | Using where |
+----+-------------+-------+------+--------------------+------+---------+-------+------+-------------+
1 row in set (0.00 sec)
0
 
LVL 30

Expert Comment

by:todd_farmer
ID: 16935922
FYI, you don't need the UNIQUE KEY, since those columns are in the primary key (which is both unique and not null).
0
 
LVL 30

Expert Comment

by:todd_farmer
ID: 16935948
Try this:

EXPLAIN SELECT * FROM table IGNORE INDEX (test) WHERE id1=10 AND num=20902 ;
0
 

Author Comment

by:pcgentry
ID: 16935951
Ok, I noticed something there... that the select statement didn't even register time-wise. As I understand it, an update with the added overhead of a write... so I ran that particular update by itself and it ran in .06 seconds, which is an acceptable time.

So I am thinking that the problem must have somehting to do with having so many statements ran one after another?
0
 

Author Comment

by:pcgentry
ID: 16935964
RE: FYI, you don't need the UNIQUE KEY, since those columns are in the primary key (which is both unique and not null).

gotcha, what can I say... I've gotten desperate :)


mysql> EXPLAIN SELECT * FROM idx IGNORE INDEX (test) WHERE mlsid=10 AND mlsnum=20902 ;
+----+-------------+-------+------+---------------+------+---------+------+-------+-------------+
| id | select_type | table | type | possible_keys | key  | key_len | ref  | rows  | Extra       |
+----+-------------+-------+------+---------------+------+---------+------+-------+-------------+
|  1 | SIMPLE      | idx   | ALL  | PRIMARY,mlsid | NULL | NULL    | NULL | 81219 | Using where |
+----+-------------+-------+------+---------------+------+---------+------+-------+-------------+
1 row in set (0.02 sec)
0
 

Author Comment

by:pcgentry
ID: 16935997
To give a better idea of what is going on here, all the sql statements are dumped into text files (currently about 10) and run with a batch file.

The largest text file has about 31 k statements in it, about 1 out of 5 similiar to the statement we have been discussing.
0
 

Author Comment

by:pcgentry
ID: 16936069
All that put aside... I ran the script again and it is moving much faster.

I guess I overworked the problem.

Thanks so much for your help!
0
 

Author Comment

by:pcgentry
ID: 16936075
or not ... it hit one of the big files and slowed down again .. and I was so excited I almost posted points :)
0
 

Author Comment

by:pcgentry
ID: 16936096
I think the statements made before were for some of the update statements that apperently run much faster..

Take a look at this:


mysql> EXPLAIN SELECT * FROM idx IGNORE INDEX (test) WHERE mlsid=22 AND mlsnum=436306 ;
+----+-------------+-------+------+---------------+------+---------+------+-------+-------------+
| id | select_type | table | type | possible_keys | key  | key_len | ref  | rows  | Extra       |
+----+-------------+-------+------+---------------+------+---------+------+-------+-------------+
|  1 | SIMPLE      | idx   | ALL  | PRIMARY,mlsid | NULL | NULL    | NULL | 81219 | Using where |
+----+-------------+-------+------+---------------+------+---------+------+-------+-------------+
1 row in set (0.28 sec)
0
 

Author Comment

by:pcgentry
ID: 16936279
mysql> EXPLAIN SELECT * FROM idx IGNORE INDEX (test) WHERE mlsid=22 AND mlsnum=436306 ;
+----+-------------+-------+------+---------------+------+---------+------+-------+-------------+
| id | select_type | table | type | possible_keys | key  | key_len | ref  | rows  | Extra       |
+----+-------------+-------+------+---------------+------+---------+------+-------+-------------+
|  1 | SIMPLE      | idx   | ALL  | PRIMARY,mlsid | NULL | NULL    | NULL | 81219 | Using where |
+----+-------------+-------+------+---------------+------+---------+------+-------+-------------+
1 row in set (1.63 sec)

mysql> EXPLAIN SELECT * FROM idx IGNORE INDEX (test) WHERE mlsid=22 AND mlsnum=436306 ;
+----+-------------+-------+------+---------------+------+---------+------+-------+-------------+
| id | select_type | table | type | possible_keys | key  | key_len | ref  | rows  | Extra       |
+----+-------------+-------+------+---------------+------+---------+------+-------+-------------+
|  1 | SIMPLE      | idx   | ALL  | PRIMARY,mlsid | NULL | NULL    | NULL | 81219 | Using where |
+----+-------------+-------+------+---------------+------+---------+------+-------+-------------+
1 row in set (1.89 sec)


Things are still much too slow... still need some help here.
0
 
LVL 30

Expert Comment

by:todd_farmer
ID: 16936391
I think that perhaps the reason it is not using the index is that your mlsnum column is defined as VARCHAR, but you're passing in a number, which MySQL converts to a VARCHAR internally.  Try this:

EXPLAIN SELECT * FROM idx IGNORE INDEX (test) WHERE mlsid=22 AND mlsnum='436306' ;
0
 

Author Comment

by:pcgentry
ID: 16936526
mysql> EXPLAIN SELECT * FROM idx IGNORE INDEX (test) WHERE mlsid=22 AND mlsnum='436306' ;
+----+-------------+-------+------+----------------------+---------+---------+-------------+------+-------------+
| id | select_type | table | type | possible_keys        | key     | key_len | ref         | rows | Extra       |
+----+-------------+-------+------+----------------------+---------+---------+-------------+------+-------------+
|  1 | SIMPLE      | idx   | ref  | PRIMARY,mlsid,mlsnum | PRIMARY | 16      | const,const |    1 | Using where |
+----+-------------+-------+------+----------------------+---------+---------+-------------+------+-------------+
1 row in set (0.42 sec)



When repeated the it ranges in time between .17 and .95 sec

0
 
LVL 30

Accepted Solution

by:
todd_farmer earned 125 total points
ID: 16936543
OK, that sounds better.  So you would want to modify your update statement to be:

UPDATE table SET thumbnailname = 'IMAGENAME.JPG' WHERE id1=10 AND num='20902' ;
0
 

Author Comment

by:pcgentry
ID: 16936831
That did it... much better!

Anything else you can think of that might speed things up?
0
 
LVL 30

Expert Comment

by:todd_farmer
ID: 16936860
If the mlsnum really is a numeric value, I would redefine the column as such and revert to the original update statement.  Comparisons using numeric values as opposed to text will be faster, but probably not by a significant measure.
0
 

Author Comment

by:pcgentry
ID: 16936918
gotcha.. can't depend on it to be numeric in any case, alpha-numerics pop up every once in awhile.


The answers you already gave have taken care of the major blockage that caused me to post. So thnx a million:)

If you can come up with anything else that may help it more, I'll be glad to send over some more points.
0

Featured Post

Upcoming Webinar: Securing your MySQL/MariaDB data

Join Percona’s Chief Evangelist, Colin Charles as he presents Securing your MySQL®/MariaDB® data on Tuesday, July 11, 2017 at 7:00 am PDT / 10:00 am EDT (UTC-7).

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

All XML, All the Time; More Fun MySQL Tidbits – Dynamically Generate XML via Stored Procedure in MySQL Extensible Markup Language (XML) and database systems, a marriage we are seeing more and more of.  So the topics of parsing and manipulating XM…
Foreword This is an old article.  Instead of using the MySQL extension that was used in the original code examples, please choose one of the currently supported database extensions instead.  More information is available here: MySQLi / PDO (http://…
NetCrunch network monitor is a highly extensive platform for network monitoring and alert generation. In this video you'll see a live demo of NetCrunch with most notable features explained in a walk-through manner. You'll also get to know the philos…
In this brief tutorial Pawel from AdRem Software explains how you can quickly find out which services are running on your network, or what are the IP addresses of servers responsible for each service. Software used is freeware NetCrunch Tools (https…

717 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question