rares_dumitrescu
asked on
freebsd / mysql 5.5 high i/o usage
hi guys,
i have a problem.
i am using mysql5.5 on a freebsd8 server, with php 5.3.13 -fpm and nginx.
when i am looking at "iostat -d 2", i see high disk i/o 20 30 mb/s in writes, however this does not translate in actual disk growth as far as i can see from monitoring "df".
i looked at "top", and hit the "m" key to toggle and see which process is causing this disk activity and it shows it's the mysql server.
i don't know where to start here, because it's a bit puzzling for me on why this high disk write doesn't actually translate in disk usage being modified as well.
can you please throw me a hint to go on the right direction?
my hardware: dual quad core xeons / 4 x 120 gb ssd's in raid10 / 16 gb ram
below is my "my.cnf" file:
i have a problem.
i am using mysql5.5 on a freebsd8 server, with php 5.3.13 -fpm and nginx.
when i am looking at "iostat -d 2", i see high disk i/o 20 30 mb/s in writes, however this does not translate in actual disk growth as far as i can see from monitoring "df".
+-(~)->iostat -d 2
aacd0 pass0 pass1 pass2
KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s
18.87 246 4.54 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00
17.94 767 13.44 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00
19.01 1032 19.16 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00
13.88 868 11.76 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00
19.04 522 9.71 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00
18.17 535 9.50 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00
19.91 2397 46.61 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00
17.99 532 9.35 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00
18.85 1291 23.76 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 i looked at "top", and hit the "m" key to toggle and see which process is causing this disk activity and it shows it's the mysql server.
i don't know where to start here, because it's a bit puzzling for me on why this high disk write doesn't actually translate in disk usage being modified as well.
can you please throw me a hint to go on the right direction?
my hardware: dual quad core xeons / 4 x 120 gb ssd's in raid10 / 16 gb ram
below is my "my.cnf" file:
[mysqld]
datadir=/var/db/mysql
socket=/tmp/mysql.sock
skip-external-locking
skip-name-resolve
query_cache_limit=256M
query_cache_size=256M
query_cache_type=1
ft_min_word_len=3
max_user_connections=200
max_connections=200
interactive_timeout=10
wait_timeout=30
connect_timeout=10
thread_cache_size=128
long_query_time=5
key_buffer_size=1024M
join_buffer=2M
max_allowed_packet=16M
table_cache=32384
sort_buffer_size=4M
read_buffer_size=4M
max_connect_errors=10
thread_concurrency=8
myisam_sort_buffer_size=64M
low_priority_updates=1
concurrent_insert=2
max_heap_table_size=128M
tmp_table_size=128M
slow_query_log=1
slow_query_log_file=/var/db/mysql/razor-slow-log-by-katmai.log
long_query_time=1
log-queries-not-using-indexes
server-id=1
local-infile=0
innodb_open_files=131072
innodb_buffer_pool_size=8192M
#new options
#innodb_log_file_size=256M
#innodb_log_buffer_size=4M
#innodb_flush_log_at_trx_commit=2
#innodb_thread_concurrency=8
############################
#for recovery
#innodb_force_recovery=4
#innodb_file_per_table=
#[mysql.server]
#user=mysql
#basedir=/var/lib
[safe_mysqld]
err-log=/var/db/mysql/razor-mysqld.log
pid-file=/var/db/mysql/mysql.pid
open_files_limit=131072
#[mysqldump]
#quick
#max_allowed_packet=16M
[mysql]
no-auto-rehash
[isamchk]
key_buffer=64M
sort_buffer=64M
read_buffer=16M
write_buffer=16M
[myisamchk]
key_buffer=64M
sort_buffer=64M
read_buffer=16M
write_buffer=16M
[mysqlhotcopy]
interactive-timeoutASKER
iostat -dx 2 output:
extended device statistics
device r/s w/s kr/s kw/s wait svc_t %b
aacd0 0.0 525.6 0.0 9413.8 0 1.4 6
pass0 0.0 0.0 0.0 0.0 0 0.0 0
pass1 0.0 0.0 0.0 0.0 0 0.0 0
pass2 0.0 0.0 0.0 0.0 0 0.0 0
pass3 0.0 0.0 0.0 0.0 0 0.0 0
pass4 0.0 0.0 0.0 0.0 0 0.0 0
extended device statistics
device r/s w/s kr/s kw/s wait svc_t %b
aacd0 0.0 476.9 0.0 9812.1 0 1.4 4
pass0 0.0 0.0 0.0 0.0 0 0.0 0
pass1 0.0 0.0 0.0 0.0 0 0.0 0
pass2 0.0 0.0 0.0 0.0 0 0.0 0
pass3 0.0 0.0 0.0 0.0 0 0.0 0
pass4 0.0 0.0 0.0 0.0 0 0.0 0
extended device statistics
device r/s w/s kr/s kw/s wait svc_t %b
aacd0 0.0 625.3 0.0 11008.4 0 1.8 3
pass0 0.0 0.0 0.0 0.0 0 0.0 0
pass1 0.0 0.0 0.0 0.0 0 0.0 0
pass2 0.0 0.0 0.0 0.0 0 0.0 0
pass3 0.0 0.0 0.0 0.0 0 0.0 0
pass4 0.0 0.0 0.0 0.0 0 0.0 0
extended device statistics
device r/s w/s kr/s kw/s wait svc_t %b
aacd0 0.0 493.8 0.0 9015.6 0 0.9 5
pass0 0.0 0.0 0.0 0.0 0 0.0 0
pass1 0.0 0.0 0.0 0.0 0 0.0 0
pass2 0.0 0.0 0.0 0.0 0 0.0 0
pass3 0.0 0.0 0.0 0.0 0 0.0 0
pass4 0.0 0.0 0.0 0.0 0 0.0 0
extended device statistics
device r/s w/s kr/s kw/s wait svc_t %b
aacd0 0.0 493.8 0.0 8951.9 0 0.9 3
pass0 0.0 0.0 0.0 0.0 0 0.0 0
pass1 0.0 0.0 0.0 0.0 0 0.0 0
pass2 0.0 0.0 0.0 0.0 0 0.0 0
pass3 0.0 0.0 0.0 0.0 0 0.0 0
pass4 0.0 0.0 0.0 0.0 0 0.0 0
extended device statistics
device r/s w/s kr/s kw/s wait svc_t %b
aacd0 0.0 459.1 0.0 8405.3 0 1.3 6
pass0 0.0 0.0 0.0 0.0 0 0.0 0
pass1 0.0 0.0 0.0 0.0 0 0.0 0
pass2 0.0 0.0 0.0 0.0 0 0.0 0
pass3 0.0 0.0 0.0 0.0 0 0.0 0
pass4 0.0 0.0 0.0 0.0 0 0.0 0 
When you catch it at high resource utilization do "show full processlist" within MySQL and take note of the actual queries and run EXPLAIN on them (see http://dev.mysql.com/doc/refman/5.5/en/using-explain.html) to see how MySQL is executing the query.
Also look at turning on the slow query log (http://dev.mysql.com/doc/refman/5.5/en/slow-query-log.html) to log long running commands (by default, long running means >2 seconds but you can tune this too). If you see non-zero values for tmp_disk_tables for long running queries, that will likely explain your high writes without growing filesystem usage (since they are deleted after). That means tmp_table_size needs some adjustment.
Also look at turning on the slow query log (http://dev.mysql.com/doc/refman/5.5/en/slow-query-log.html) to log long running commands (by default, long running means >2 seconds but you can tune this too). If you see non-zero values for tmp_disk_tables for long running queries, that will likely explain your high writes without growing filesystem usage (since they are deleted after). That means tmp_table_size needs some adjustment.
ASKER
thats the thing. the i/o is constant, and the sql has been optimized pretty much. there are no slow queries, as i already have the slow query log enabled.
i have been running mysqltuner and tuned 2 3 extra parameters and the final result is the my.cnf above, where you can see i already have the slow log enabled.
->mysqladmin processlist
+---------+-------+--------------------+-------+---------+------+----------------+------------------+
| Id | User | Host | db | Command | Time | State | Info |
+---------+-------+--------------------+-------+---------+------+----------------+------------------+
| 2290730 | magic | 1.2.3.4:65245 | magic | Sleep | 1 | | |
| 2294828 | magic | localhost | magic | Sleep | 17 | | |
| 2295124 | magic | 1.2.3.4:10112 | magic | Sleep | 0 | | |
| 2295623 | magic | 1.2.3.4:10173 | magic | Sleep | 1 | | |
| 2296687 | magic | localhost | magic | Sleep | 2 | | |
| 2296728 | magic | 1.2.3.4:10305 | magic | Sleep | 1 | | |
| 2296792 | magic | localhost | magic | Sleep | 1 | | |
| 2296803 | magic | localhost | magic | Sleep | 1 | | |
| 2296816 | magic | 1.2.3.4:10319 | magic | Sleep | 0 | | |
| 2296818 | root | localhost | | Query | 0 | | show processlist |
| 2296819 | magic | localhost | magic | Sleep | 0 | | |
| 2296820 | magic | localhost | magic | Connect | | Writing to net | |
+---------+-------+--------------------+-------+---------+------+----------------+------------------+i have been running mysqltuner and tuned 2 3 extra parameters and the final result is the my.cnf above, where you can see i already have the slow log enabled.
-------- Performance Metrics -------------------------------------------------
[--] Up for: 4h 23m 37s (12M q [816.093 qps], 2M conn, TX: 16B, RX: 1B)
[--] Reads / Writes: 82% / 18%
[--] Total buffers: 9.4G global + 10.5M per thread (200 max threads)
[OK] Maximum possible memory usage: 11.4G (67% of installed RAM)
[OK] Slow queries: 0% (1K/12M)
[OK] Highest usage of available connections: 32% (65/200)
[OK] Key buffer size / total MyISAM indexes: 1.0G/2.3G
[OK] Key buffer hit rate: 99.8% (84M cached / 180K reads)
[OK] Query cache efficiency: 36.4% (3M cached / 9M selects)
[OK] Query cache prunes per day: 0
[OK] Sorts requiring temporary tables: 0% (62 temp sorts / 1M sorts)
[OK] Temporary tables created on disk: 9% (66K on disk / 732K total)
[OK] Thread cache hit rate: 99% (65 created / 2M connections)
[OK] Table cache hit rate: 69% (356 open / 514 opened)
[OK] Open file limit used: 0% (277/131K)
[OK] Table locks acquired immediately: 99% (9M immediate / 9M locks)
[OK] InnoDB data size / buffer pool: 3.1G/8.0G
-------- Recommendations -----------------------------------------------------
Yeah sorry I missed that part about your my.cnf. Now I'm dumbfounded myself. Better call in other experts.
ASKER
waiting for some guys that have any ideas then.
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
run top command how much free ram you have? how much swap is used?
Run this scripts:
http://www.freebsd.org/cgi/cvsweb.cgi/ports/databases/mysqltuner/
http://www.freebsd.org/cgi/cvsweb.cgi/ports/databases/tuning-primer/
Run this scripts:
http://www.freebsd.org/cgi/cvsweb.cgi/ports/databases/mysqltuner/
http://www.freebsd.org/cgi/cvsweb.cgi/ports/databases/tuning-primer/
ASKER
doesnt seem to use the swap too much here...
204 processes: 1 running, 203 sleeping
CPU: 16.2% user, 0.0% nice, 1.1% system, 0.2% interrupt, 82.5% idle
Mem: 6091M Active, 6550M Inact, 2347M Wired, 303M Cache, 1646M Buf, 558M Free
Swap: 8000M Total, 48K Used, 8000M Free
204 processes: 1 running, 203 sleeping
CPU: 16.2% user, 0.0% nice, 1.1% system, 0.2% interrupt, 82.5% idle
Mem: 6091M Active, 6550M Inact, 2347M Wired, 303M Cache, 1646M Buf, 558M Free
Swap: 8000M Total, 48K Used, 8000M Free
ASKER
i optimized a bunch of db parameters and code based on tuning primer and xhprofiler.
/usr/ports/databases/tuning-primer)->tuning-primer
-- MYSQL PERFORMANCE TUNING PRIMER --
- By: Matthew Montgomery -
MySQL Version 5.5.25-log amd64
Uptime = 0 days 9 hrs 55 min 21 sec
Avg. qps = 648
Total Questions = 23150780
Threads Connected = 14
Warning: Server has not been running for at least 48hrs.
It may not be safe to use these recommendations
To find out more information on how each of these
runtime variables effects performance visit:
http://dev.mysql.com/doc/refman/5.5/en/server-system-variables.html
Visit http://www.mysql.com/products/enterprise/advisors.html
for info about MySQL's Enterprise Monitoring and Advisory Service
SLOW QUERIES
The slow query log is enabled.
Current long_query_time = 1.000000 sec.
You have 2307 out of 23150871 that take longer than 1.000000 sec. to complete
Your long_query_time seems to be fine
BINARY UPDATE LOG
The binary update log is NOT enabled.
You will not be able to do point in time recovery
See http://dev.mysql.com/doc/refman/5.5/en/point-in-time-recovery.html
WORKER THREADS
Current thread_cache_size = 128
Current threads_cached = 58
Current threads_per_sec = 0
Historic threads_per_sec = 0
Your thread_cache_size is fine
MAX CONNECTIONS
Current max_connections = 200
Current threads_connected = 6
Historic max_used_connections = 65
The number of used connections is 32% of the configured maximum.
Your max_connections variable seems to be fine.
INNODB STATUS
Current InnoDB index space = 2.25 G
Current InnoDB data space = 3.13 G
Current InnoDB buffer pool free = 47 %
Current innodb_buffer_pool_size = 8.00 G
Depending on how much space your innodb indexes take up it may be safe
to increase this value to up to 2 / 3 of total system memory
MEMORY USAGE
Max Memory Ever Allocated : 9.93 G
Configured Max Per-thread Buffers : 2.05 G
Configured Max Global Buffers : 9.26 G
Configured Max Memory Limit : 11.31 G
Physical Memory : 17.00 G
Max memory limit seem to be within acceptable norms
KEY BUFFER
Current MyISAM index space = 2.28 G
Current key_buffer_size = 1.00 G
Key cache miss rate is 1 : 565
Key buffer free ratio = 60 %
Your key_buffer_size seems to be fine
QUERY CACHE
Query cache is enabled
Current query_cache_size = 256 M
Current query_cache_used = 4 M
Current query_cache_limit = 256 M
Current Query cache Memory fill ratio = 1.61 %
Current query_cache_min_res_unit = 4 K
Query Cache is 29 % fragmented
Run "FLUSH QUERY CACHE" periodically to defragment the query cache memory
If you have many small queries lower 'query_cache_min_res_unit' to reduce fragmentation.
Your query_cache_size seems to be too high.
Perhaps you can use these resources elsewhere
MySQL won't cache query results that are larger than query_cache_limit in size
SORT OPERATIONS
Current sort_buffer_size = 4 M
Current read_rnd_buffer_size = 256 K
Sort buffer seems to be fine
JOINS
Current join_buffer_size = 2.00 M
You have had 0 queries where a join could not use an index properly
Your joins seem to be using indexes properly
OPEN FILES LIMIT
Current open_files_limit = 131072 files
The open_files_limit should typically be set to at least 2x-3x
that of table_cache if you have heavy MyISAM usage.
Your open_files_limit value seems to be fine
TABLE CACHE
Current table_open_cache = 32384 tables
Current table_definition_cache = 400 tables
You have a total of 152 tables
You have 357 open tables.
The table_cache value seems to be fine
TEMP TABLES
Current max_heap_table_size = 128 M
Current tmp_table_size = 128 M
Of 1196033 temp tables, 8% were created on disk
Created disk tmp tables ratio seems fine
TABLE SCANS
Current read_buffer_size = 4 M
Current table scan ratio = 22 : 1
read_buffer_size seems to be fine
TABLE LOCKING
Current Lock Wait ratio = 1 : 49874
Your table locking seems to be fine
Swap: 8000M Total, 48K Used, 8000M Free << Does this mean it is already allocated on disk; if so, then if data is written into swap, you will not see growth but there will be I/O.
Anyway, please check on the database free space. That is probably the likelier issue given you seem to have memory and configurations to cover MySQL queries.
Anyway, please check on the database free space. That is probably the likelier issue given you seem to have memory and configurations to cover MySQL queries.
ASKER
i got 4 x 120 gb drives in raid10. ssd's
there's no disk space overload by any means:
there's no disk space overload by any means:
+-(/usr/ports/databases/tuning-primer)->df -h
Filesystem Size Used Avail Capacity Mounted on
/dev/aacd0s1a 197G 65G 117G 36% /
devfs 1.0K 1.0K 0B 100% /dev
/dev/aacd0s1d 11G 104M 10G 1% /tmp
linprocfs 4.0K 4.0K 0B 100% /usr/compat/linux/procASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
sorry about that.
mysql> SHOW TABLE STATUS
-> ;
+-----------------+--------+---------+------------+---------+----------------+-------------+------------------+--------------+-----------+----------------+---------------------+---------------------+---------------------+-------------------+----------+----------------+---------+
| Name | Engine | Version | Row_format | Rows | Avg_row_length | Data_length | Max_data_length | Index_length | Data_free | Auto_increment | Create_time | Update_time | Check_time | Collation | Checksum | Create_options | Comment |
+-----------------+--------+---------+------------+---------+----------------+-------------+------------------+--------------+-----------+----------------+---------------------+---------------------+---------------------+-------------------+----------+----------------+---------+
| admin | MyISAM | 10 | Fixed | 3 | 9 | 27 | 2533274790395903 | 2048 | 0 | 5 | 2012-04-06 15:16:44 | 2012-04-06 15:16:44 | NULL | latin1_swedish_ci | NULL | | |
| api_token | InnoDB | 10 | Compact | 0 | 0 | 16384 | 0 | 0 | 5242880 | NULL | 2012-06-18 03:01:23 | NULL | NULL | utf8_general_ci | NULL | | |
| avatars | MyISAM | 10 | Dynamic | 147716 | 67 | 10044408 | 281474976710655 | 3424256 | 0 | 147718 | 2012-04-06 15:16:44 | 2012-06-21 02:41:10 | 2012-04-06 15:16:45 | latin1_swedish_ci | NULL | | |
| block_user | InnoDB | 10 | Compact | 91754 | 40 | 3686400 | 0 | 7012352 | 5242880 | 122534 | 2012-06-18 13:17:18 | NULL | NULL | latin1_swedish_ci | NULL | | |
| comments | MyISAM | 10 | Dynamic | 4176533 | 94 | 454181912 | 281474976710655 | 197377024 | 57535616 | 4502643 | 2012-04-06 15:16:54 | 2012-06-21 02:42:57 | 2012-04-06 15:17:32 | latin1_swedish_ci | NULL | | |
| conversations | InnoDB | 10 | Compact | 1030444 | 49 | 50937856 | 0 | 23642112 | 5242880 | NULL | 2012-06-19 06:08:22 | NULL | NULL | latin1_swedish_ci | NULL | | |
| convert_queue | MyISAM | 10 | Dynamic | 42167 | 254 | 10746936 | 281474976710655 | 1104896 | 0 | 52434 | 2012-06-18 03:02:52 | 2012-06-21 02:42:57 | NULL | latin1_swedish_ci | NULL | | |
| country | MyISAM | 10 | Dynamic | 215 | 25 | 5512 | 281474976710655 | 7168 | 0 | 223 | 2012-04-06 15:17:40 | 2012-04-06 15:17:40 | 2012-04-06 15:17:40 | latin1_swedish_ci | NULL | | |
| email_queue | MyISAM | 10 | Dynamic | 0 | 0 | 228633856 | 281474976710655 | 52898816 | 228633856 | 2206413 | 2012-06-18 00:06:01 | 2012-06-20 00:08:05 | NULL | latin1_swedish_ci | NULL | | |
| facebook_action | InnoDB | 10 | Compact | 8926 | 178 | 1589248 | 0 | 0 | 5242880 | NULL | 2012-05-28 04:19:12 | NULL | NULL | utf8_general_ci | NULL | | |
| featured | MyISAM | 10 | Fixed | 196 | 13 | 2652 | 3659174697238527 | 9216 | 104 | 250 | 2012-04-06 15:18:08 | 2012-06-13 02:28:16 | NULL | latin1_swedish_ci | NULL | | |
| followers | MyISAM | 10 | Fixed | 2861960 | 17 | 48653337 | 4785074604081151 | 167976960 | 17 | 3980442 | 2012-04-06 15:18:08 | 2012-06-21 02:42:57 | 2012-04-06 15:18:24 | latin1_swedish_ci | NULL | | |
| homepage_grid | MyISAM | 10 | Dynamic | 24 | 99 | 2392 | 281474976710655 | 2048 | 0 | NULL | 2012-04-06 15:18:24 | 2012-06-18 12:48:34 | NULL | latin1_swedish_ci | NULL | | |
| likes | MyISAM | 10 | Fixed | 851136 | 17 | 14469397 | 4785074604081151 | 34427904 | 85 | 20843345 | 2012-06-18 05:51:12 | 2012-06-21 02:42:44 | 2012-06-18 05:51:19 | latin1_swedish_ci | NULL | | |
| message_archive | InnoDB | 10 | Compact | 17959 | 83 | 1490944 | 0 | 425984 | 5242880 | NULL | 2012-06-18 13:28:55 | NULL | NULL | latin1_swedish_ci | NULL | | |
| messages | InnoDB | 10 | Compact | 5136949 | 276 | 1421852672 | 0 | 803766272 | 5242880 | 10754152 | 2012-06-18 03:45:00 | NULL | NULL | latin1_swedish_ci | NULL | | |
| notification | InnoDB | 10 | Compact | 6391581 | 142 | 908017664 | 0 | 1106575360 | 5242880 | 11970019 | 2012-06-18 14:09:46 | NULL | NULL | utf8_general_ci | NULL | | |
| oauth_provider | InnoDB | 10 | Compact | 19438 | 458 | 8912896 | 0 | 1589248 | 5242880 | 478238 | 2012-06-18 13:36:03 | NULL | NULL | latin1_swedish_ci | NULL | | |
| picture_views | InnoDB | 10 | Compact | 811984 | 48 | 39387136 | 0 | 44122112 | 5242880 | NULL | 2012-06-18 03:59:15 | NULL | NULL | utf8_general_ci | NULL | | |
| pictures | InnoDB | 10 | Compact | 2912015 | 133 | 387825664 | 0 | 194281472 | 5242880 | 4218005 | 2012-06-18 04:04:57 | NULL | NULL | latin1_swedish_ci | NULL | | |
| report | MyISAM | 10 | Dynamic | 0 | 0 | 0 | 281474976710655 | 1024 | 0 | 1 | 2012-04-06 15:21:57 | 2012-04-06 15:21:57 | NULL | latin1_swedish_ci | NULL | | |
| settings | MyISAM | 10 | Fixed | 1 | 7 | 7 | 1970324836974591 | 2048 | 0 | 3 | 2012-04-06 15:21:57 | 2012-04-06 15:21:57 | NULL | latin1_swedish_ci | NULL | | |
| static_pages | MyISAM | 10 | Dynamic | 6 | 7809 | 46856 | 281474976710655 | 2048 | 0 | 7 | 2012-04-06 15:21:57 | 2012-06-18 07:54:45 | NULL | latin1_swedish_ci | NULL | | |
| stats | MyISAM | 10 | Fixed | 218 | 15 | 3300 | 4222124650659839 | 9216 | 30 | 6599 | 2012-04-06 15:21:57 | 2012-06-21 00:29:54 | NULL | latin1_swedish_ci | NULL | | |
| trending | InnoDB | 10 | Compact | 72 | 227 | 16384 | 0 | 16384 | 5242880 | 94 | 2012-06-18 05:50:00 | NULL | NULL | utf8_general_ci | NULL | | |
| user_ip | MyISAM | 10 | Dynamic | 1484964 | 25 | 37558564 | 281474976710655 | 35785728 | 0 | NULL | 2012-04-06 15:21:57 | 2012-06-21 02:42:57 | NULL | latin1_swedish_ci | NULL | | |
| user_ip_ban | MyISAM | 10 | Fixed | 444 | 9 | 3996 | 2533274790395903 | 7168 | 0 | NULL | 2012-04-06 15:22:10 | 2012-06-21 01:21:15 | NULL | latin1_swedish_ci | NULL | | |
| users | InnoDB | 10 | Compact | 484353 | 202 | 98156544 | 0 | 60473344 | 5242880 | 478238 | 2012-06-18 13:07:05 | NULL | NULL | latin1_swedish_ci | NULL | | |
| vote_history | InnoDB | 10 | Compact | 0 | 0 | 16384 | 0 | 0 | 5242880 | NULL | 2012-06-18 04:00:14 | NULL | NULL | utf8_general_ci | NULL | | |
+-----------------+--------+---------+------------+---------+----------------+-------------+------------------+--------------+-----------+----------------+---------------------+---------------------+---------------------+-------------------+----------+----------------+---------+
29 rows in set (0.05 sec)
Look at the Data_free column. You have about 400mb of allocated but unused space. Is it possible that your application is constantly deleting records as well as writing them? What you may be seeing is the fact that the writes can fit within the unused space for each table because of the rate of deletes. To check this, after a period of writes, check to see if the values in Data_free have changed any--or any of the stats in SHOW TABLE STATUS for that matter.
ASKER
actually we loaded some more data in memcached, and cut out the i/o by a LOT. thanks for the help guys!
ASKER
Open in new window