'Copying to Tmp Table' in mysql after converting to InnoDB


I have converted my main table to Innodb from MyIsam to improve db speed (locking issues).
Since doing so my macing is grinding to a halt, when looking at the connections most of then seem to say 'Copying to tmp table'. Im assuming this is a disk based table and as a result is slowing down the process. I cant understand why the Server is doing this.

Is is a memory allocation issue? (I have reinstalled my server with a high volume setting)
A Query issue? (It happens on most queries, but didnt when I used a MyIsam table)

I have searched google to see why 'copying to tmp table' happens but have not got any answers.

Please help..
My servers dying!
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Aleksandar BradarićSoftware DeveloperCommented:
> most of then seem to say 'Copying to tmp table'. Im assuming this is a disk based table and as a result is slowing down the process.
> I cant understand why the Server is doing this.

The server gets into the `copying to tmp table` message when MySQL is manipulating the data. If it takes too long, that usually means that the query is too slow (e.g. not having correct indexes, etc). Try checking your queries with EXPLAIN SELECT to see how they use your indexes.
UmeshSenior Principal Technical Support EngineerCommented:
"Copying to tmp table" - The server is copying to a temporary table in memory. .. I'm agree with  "leannonn"  on tuning queries at first place.

Please monitor these (if not enabled pls do so) -- need to set in my.cnf
1. Queries not using indexes
2. Slow queries

Also could you brief me about
- is this serve dedicated to MySQL?
- How much InnoDB/MyISAM tables used?
- post your my.cnf
- post output of below statements
  show global status;
  show global variables;


jellydealAuthor Commented:
Hi Ushastry,

The machine is running IIS and MySQL 5 server. I am serving one website that does get quite a lot of traffic. Approx 300,000 pages (majority dadabase driven).

Here is the info you asked for, hope you can shine a light on it..

thanks agin,

CPU INTEL 6600 * 2 @ 2.6Ghz
MEM 2 Gig
OS  Win 2003 Server
MYSQL V5.1.11

MYISAM 1 Table - total 500k - index 85k
InnoDB 40 Tables - total 800 Meg - index 2 Gig
# MySQL Server Instance Configuration File
# ----------------------------------------------------------------------
# Generated by the MySQL Server Instance Configuration Wizard
# Installation Instructions
# ----------------------------------------------------------------------
# On Linux you can copy this file to /etc/my.cnf to set global options,
# mysql-data-dir/my.cnf to set server-specific options
# (@localstatedir@ for this installation) or to
# ~/.my.cnf to set user-specific options.
# On Windows you should keep this file in the installation directory
# of your server (e.g. C:\Program Files\MySQL\MySQL Server X.Y). To
# make sure the server reads the config file use the startup option
# "--defaults-file".
# To run run the server from the command line, execute this in a
# command line shell, e.g.
# mysqld --defaults-file="C:\Program Files\MySQL\MySQL Server X.Y\my.ini"
# To install the server as a Windows service manually, execute this in a
# command line shell, e.g.
# mysqld --install MySQLXY --defaults-file="C:\Program Files\MySQL\MySQL Server X.Y\my.ini"
# And then execute this in a command line shell to start the server, e.g.
# net start MySQLXY
# Guildlines for editing this file
# ----------------------------------------------------------------------
# In this file, you can use all long options that the program supports.
# If you want to know the options a program supports, start the program
# with the "--help" option.
# More detailed information about the individual options can also be
# found in the manual.
# ----------------------------------------------------------------------
# The following options will be read by MySQL client applications.
# Note that only client applications shipped by MySQL are guaranteed
# to read this section. If you want your own MySQL client program to
# honor these values, you need to specify it as an option during the
# MySQL client library initialization.




# ----------------------------------------------------------------------
# The following options will be read by the MySQL Server. Make sure that
# you have installed the server correctly (see above) so it reads this
# file.

# The TCP/IP Port the MySQL Server will listen on

#Path to installation directory. All paths are usually resolved relative to this.
basedir="E:/Program Files/MySQL/MySQL Server 5.0_INNODB/"

#Path to the database root
datadir="E:/Program Files/MySQL/MySQL Server 5.0_INNODB/Data/"

# The default character set that will be used when a new schema or table is
# created and no character set is defined

# The default storage engine that will be used when create new tables when

# Set the SQL mode to strict

# The maximum amount of concurrent sessions the MySQL server will
# allow. One of these connections will be reserved for a user with
# SUPER privileges to allow the administrator to login even if the
# connection limit has been reached.

# Query cache is used to cache SELECT results and later return them
# without actual executing the same query once again. Having the query
# cache enabled may result in significant speed improvements, if your
# have a lot of identical queries and rarely changing tables. See the
# "Qcache_lowmem_prunes" status variable to check if the current value
# is high enough for your load.
# Note: In case your tables change very often or if your queries are
# textually different every time, the query cache may result in a
# slowdown instead of a performance improvement.

# The number of open tables for all threads. Increasing this value
# increases the number of file descriptors that mysqld requires.
# Therefore you have to make sure to set the amount of open files
# allowed to at least 4096 in the variable "open-files-limit" in
# section [mysqld_safe]

# Maximum size for internal (in-memory) temporary tables. If a table
# grows larger than this value, it is automatically converted to disk
# based table This limitation is for a single table. There can be many
# of them.

# How many threads we should keep in a cache for reuse. When a client
# disconnects, the client's threads are put in the cache if there aren't
# more than thread_cache_size threads from before.  This greatly reduces
# the amount of thread creations needed if you have a lot of new
# connections. (Normally this doesn't give a notable performance
# improvement if you have a good thread implementation.)

#*** MyISAM Specific options

# The maximum size of the temporary file MySQL is allowed to use while
# recreating the index (during REPAIR, ALTER TABLE or LOAD DATA INFILE.
# If the file-size would be bigger than this, the index will be created
# through the key cache (which is slower).

# If the temporary file used for fast index creation would be bigger
# than using the key cache by the amount specified here, then prefer the
# key cache method.  This is mainly used to force long character keys in
# large tables to use the slower key cache method to create the index.

# If the temporary file used for fast index creation would be bigger
# than using the key cache by the amount specified here, then prefer the
# key cache method.  This is mainly used to force long character keys in
# large tables to use the slower key cache method to create the index.

# Size of the Key Buffer, used to cache index blocks for MyISAM tables.
# Do not set it larger than 30% of your available memory, as some memory
# is also required by the OS to cache rows. Even if you're not using
# MyISAM tables, you should still set it to 8-64M as it will also be
# used for internal temporary disk tables.

# Size of the buffer used for doing full table scans of MyISAM tables.
# Allocated per thread, if a full scan is needed.

# This buffer is allocated when MySQL needs to rebuild the index in
# REPAIR, OPTIMZE, ALTER table statements as well as in LOAD DATA INFILE
# into an empty table. It is allocated per thread so be careful with
# large settings.

#*** INNODB Specific options ***

# Use this option if you have a MySQL server with InnoDB support enabled
# but you do not plan to use it. This will save memory and disk space
# and speed up some things.

# Additional memory pool that is used by InnoDB to store metadata
# information.  If InnoDB requires more memory for this purpose it will
# start to allocate it from the OS.  As this is fast enough on most
# recent operating systems, you normally do not need to change this
# value. SHOW INNODB STATUS will display the current amount used.

# If set to 1, InnoDB will flush (fsync) the transaction logs to the
# disk at each commit, which offers full ACID behavior. If you are
# willing to compromise this safety, and you are running small
# transactions, you may set this to 0 or 2 to reduce disk I/O to the
# logs. Value 0 means that the log is only written to the log file and
# the log file flushed to disk approximately once per second. Value 2
# means the log is written to the log file at each commit, but the log
# file is only flushed to disk approximately once per second.

# The size of the buffer InnoDB uses for buffering log data. As soon as
# it is full, InnoDB will have to flush it to disk. As it is flushed
# once per second anyway, it does not make sense to have it very large
# (even with long transactions).

# InnoDB, unlike MyISAM, uses a buffer pool to cache both indexes and
# row data. The bigger you set this the less disk I/O is needed to
# access data in tables. On a dedicated database server you may set this
# parameter up to 80% of the machine physical memory size. Do not set it
# too large, though, because competition of the physical memory may
# cause paging in the operating system.  Note that on 32bit systems you
# might be limited to 2-3.5G of user level memory per process, so do not
# set it too high.

# Size of each log file in a log group. You should set the combined size
# of log files to about 25%-100% of your buffer pool size to avoid
# unneeded buffer pool flush activity on log file overwrite. However,
# note that a larger logfile size will increase the time needed for the
# recovery process.

# Number of threads allowed inside the InnoDB kernel. The optimal value
# depends highly on the application, hardware as well as the OS
# scheduler properties. A too high value may lead to thread thrashing.
#Enter a name for the slow query log. Otherwise a default name will be used.
#Enter a name for the error log file. Otherwise a default name will be used.
#Method used for flushing the log files to disk.
#Enter a name for the query log file. Otherwise a default name will be used.
#Path to the temporary directory.

Aborted_clients 48
Aborted_connects 250
Binlog_cache_disk_use 0
Binlog_cache_use 0
Bytes_received 323668920
Bytes_sent 1655965428
Com_admin_commands 1078307
Com_alter_db 0
Com_alter_table 0
Com_analyze 0
Com_backup_table 0
Com_begin 1
Com_call_procedure 0
Com_change_db 981452
Com_change_master 0
Com_check 0
Com_checksum 0
Com_commit 1
Com_create_db 0
Com_create_function 0
Com_create_index 0
Com_create_table 0
Com_create_user 0
Com_dealloc_sql 0
Com_delete 4
Com_delete_multi 0
Com_do 0
Com_drop_db 0
Com_drop_function 0
Com_drop_index 0
Com_drop_table 0
Com_drop_user 0
Com_execute_sql 0
Com_flush 0
Com_grant 0
Com_ha_close 0
Com_ha_open 0
Com_ha_read 0
Com_help 0
Com_insert 53311
Com_insert_select 0
Com_kill 0
Com_load 0
Com_load_master_data 0
Com_load_master_table 0
Com_lock_tables 0
Com_optimize 0
Com_preload_keys 0
Com_prepare_sql 0
Com_purge 0
Com_purge_before_date 0
Com_rename_table 0
Com_repair 0
Com_replace 0
Com_replace_select 0
Com_reset 0
Com_restore_table 0
Com_revoke 0
Com_revoke_all 0
Com_rollback 0
Com_savepoint 0
Com_select 342243
Com_set_option 6797
Com_show_binlog_events 0
Com_show_binlogs 0
Com_show_charsets 0
Com_show_collations 0
Com_show_column_types 0
Com_show_create_db 0
Com_show_create_table 75
Com_show_databases 7
Com_show_errors 0
Com_show_fields 210
Com_show_grants 0
Com_show_innodb_status 79988
Com_show_keys 80
Com_show_logs 0
Com_show_master_status 0
Com_show_ndb_status 0
Com_show_new_master 0
Com_show_open_tables 0
Com_show_privileges 0
Com_show_processlist 45
Com_show_slave_hosts 0
Com_show_slave_status 0
Com_show_status 79990
Com_show_storage_engines 0
Com_show_tables 8
Com_show_triggers 0
Com_show_variables 0
Com_show_warnings 0
Com_slave_start 0
Com_slave_stop 0
Com_stmt_close 0
Com_stmt_execute 0
Com_stmt_fetch 0
Com_stmt_prepare 0
Com_stmt_reset 0
Com_stmt_send_long_data 0
Com_truncate 0
Com_unlock_tables 0
Com_update 120112
Com_update_multi 0
Com_xa_commit 0
Com_xa_end 0
Com_xa_prepare 0
Com_xa_recover 0
Com_xa_rollback 0
Com_xa_start 0
Compression OFF
Connections 7017
Created_tmp_disk_tables 55378
Created_tmp_files 2017
Created_tmp_tables 157907
Delayed_errors 0
Delayed_insert_threads 0
Delayed_writes 0
Flush_commands 1
Handler_commit 153990
Handler_delete 0
Handler_discover 0
Handler_prepare 0
Handler_read_first 77677
Handler_read_key 24832620
Handler_read_next 1885573849
Handler_read_prev 273389
Handler_read_rnd 1305584
Handler_read_rnd_next 2173581452
Handler_rollback 18311
Handler_savepoint 0
Handler_savepoint_rollback 0
Handler_update 1819245
Handler_write 38158177
Innodb_buffer_pool_pages_data 61823
Innodb_buffer_pool_pages_dirty 13
Innodb_buffer_pool_pages_flushed 237759
Innodb_buffer_pool_pages_free 0
Innodb_buffer_pool_pages_latched 0
Innodb_buffer_pool_pages_misc 2177
Innodb_buffer_pool_pages_total 64000
Innodb_buffer_pool_read_ahead_rnd 3606
Innodb_buffer_pool_read_ahead_seq 6126
Innodb_buffer_pool_read_requests 2978547405
Innodb_buffer_pool_reads 321587
Innodb_buffer_pool_wait_free 0
Innodb_buffer_pool_write_requests 7019002
Innodb_data_fsyncs 63986
Innodb_data_pending_fsyncs 0
Innodb_data_pending_reads 0
Innodb_data_pending_writes 0
Innodb_data_read 3303428096
Innodb_data_reads 725790
Innodb_data_writes 291399
Innodb_data_written 2083147776
Innodb_dblwr_pages_written 237759
Innodb_dblwr_writes 9680
Innodb_log_waits 200
Innodb_log_write_requests 9331274
Innodb_log_writes 35336
Innodb_os_log_fsyncs 42455
Innodb_os_log_pending_fsyncs 0
Innodb_os_log_pending_writes 0
Innodb_os_log_written 2878496768
Innodb_page_size 16384
Innodb_pages_created 18642
Innodb_pages_read 725780
Innodb_pages_written 237759
Innodb_row_lock_current_waits 0
Innodb_row_lock_time 268312
Innodb_row_lock_time_avg 49
Innodb_row_lock_time_max 937
Innodb_row_lock_waits 5385
Innodb_rows_deleted 1
Innodb_rows_inserted 35000
Innodb_rows_read 3971156358
Innodb_rows_updated 3030499
Key_blocks_not_flushed 0
Key_blocks_unused 134386
Key_blocks_used 5763
Key_read_requests 187282355
Key_reads 107419
Key_write_requests 53507
Key_writes 0
Last_query_cost 0.000000
Max_used_connections 51
Not_flushed_delayed_rows 0
Open_files 12
Open_streams 0
Open_tables 62
Opened_tables 1735
Prepared_stmt_count 0
Qcache_free_blocks 847
Qcache_free_memory 3284656
Qcache_hits 631960
Qcache_inserts 296965
Qcache_lowmem_prunes 200332
Qcache_not_cached 125662
Qcache_queries_in_cache 18264
Qcache_total_blocks 39140
Questions 2341140
Rpl_status NULL
Select_full_join 2
Select_full_range_join 0
Select_range 9293
Select_range_check 0
Select_scan 145384
Slave_open_temp_tables 0
Slave_retried_transactions 0
Slave_running OFF
Slow_launch_threads 0
Slow_queries 147
Sort_merge_passes 1193
Sort_range 128392
Sort_rows 11602569
Sort_scan 78306
Ssl_accept_renegotiates 0
Ssl_accepts 0
Ssl_callback_cache_hits 0
Ssl_cipher (null)
Ssl_cipher_list (null)
Ssl_client_connects 0
Ssl_connect_renegotiates 0
Ssl_ctx_verify_depth 0
Ssl_ctx_verify_mode 0
Ssl_default_timeout 0
Ssl_finished_accepts 0
Ssl_finished_connects 0
Ssl_session_cache_hits 0
Ssl_session_cache_misses 0
Ssl_session_cache_mode NONE
Ssl_session_cache_overflows 0
Ssl_session_cache_size 0
Ssl_session_cache_timeouts 0
Ssl_sessions_reused 0
Ssl_used_session_cache_entries 0
Ssl_verify_depth 0
Ssl_verify_mode 0
Ssl_version (null)
Table_locks_immediate 893640
Table_locks_waited 13
Tc_log_max_pages_used 0
Tc_log_page_size 0
Tc_log_page_waits 0
Threads_cached 22
Threads_connected 28
Threads_created 300
Threads_running 1
Uptime 79025
Uptime_since_flush_status 79025
auto_increment_increment 1
auto_increment_offset 1
automatic_sp_privileges ON
back_log 50
basedir E:\Program Files\MySQL\MySQL Server 5.0_INNODB\
binlog_cache_size 32768
bulk_insert_buffer_size 8388608
character_set_client latin1
character_set_connection latin1
character_set_database latin1
character_set_filesystem binary
character_set_results latin1
character_set_server latin1
character_set_system utf8
character_sets_dir E:\Program Files\MySQL\MySQL Server 5.0_INNODB\share\charsets\
collation_connection latin1_swedish_ci
collation_database latin1_swedish_ci
collation_server latin1_swedish_ci
completion_type 0
concurrent_insert 1
connect_timeout 5
datadir E:\Program Files\MySQL\MySQL Server 5.0_INNODB\Data\
date_format %Y-%m-%d
datetime_format %Y-%m-%d %H:%i:%s
default_week_format 0
delay_key_write ON
delayed_insert_limit 100
delayed_insert_timeout 300
delayed_queue_size 1000
div_precision_increment 4
engine_condition_pushdown OFF
expire_logs_days 0
flush OFF
flush_time 1800
ft_boolean_syntax + -><()~*:""&|
ft_max_word_len 84
ft_min_word_len 4
ft_query_expansion_limit 20
ft_stopword_file (built-in)
group_concat_max_len 1024
have_archive YES
have_bdb NO
have_blackhole_engine YES
have_compress YES
have_crypt NO
have_csv NO
have_dynamic_loading YES
have_example_engine NO
have_federated_engine YES
have_geometry YES
have_innodb YES
have_isam NO
have_merge_engine YES
have_ndbcluster NO
have_openssl DISABLED
have_ssl DISABLED
have_query_cache YES
have_raid NO
have_rtree_keys YES
have_symlink YES
hostname DSVR005889
init_connect (null)
init_file (null)
init_slave (null)
innodb_additional_mem_pool_size 20971520
innodb_autoextend_increment 8
innodb_buffer_pool_awe_mem_mb 0
innodb_buffer_pool_size 1048576000
innodb_checksums ON
innodb_commit_concurrency 0
innodb_concurrency_tickets 500
innodb_data_file_path ibdata1:10M:autoextend
innodb_data_home_dir (null)
innodb_doublewrite ON
innodb_fast_shutdown 1
innodb_file_io_threads 4
innodb_file_per_table OFF
innodb_flush_log_at_trx_commit 0
innodb_flush_method (null)
innodb_force_recovery 0
innodb_lock_wait_timeout 50
innodb_locks_unsafe_for_binlog OFF
innodb_log_arch_dir (null)
innodb_log_archive OFF
innodb_log_buffer_size 8388608
innodb_log_file_size 268435456
innodb_log_files_in_group 2
innodb_log_group_home_dir .\
innodb_max_dirty_pages_pct 90
innodb_max_purge_lag 0
innodb_mirrored_log_groups 1
innodb_open_files 300
innodb_rollback_on_timeout OFF
innodb_support_xa ON
innodb_sync_spin_loops 20
innodb_table_locks ON
innodb_thread_concurrency 8
innodb_thread_sleep_delay 10000
interactive_timeout 28800
join_buffer_size 131072
key_buffer_size 157286400
key_cache_age_threshold 300
key_cache_block_size 1024
key_cache_division_limit 100
language E:\Program Files\MySQL\MySQL Server 5.0_INNODB\share\english\
large_files_support ON
large_page_size 0
large_pages OFF
lc_time_names en_US
license GPL
local_infile ON
log OFF
log_bin OFF
log_bin_trust_function_creators OFF
log_error E:\Program Files\MySQL\MySQL Server 5.0_INNODB\Data\DSVR005889.err
log_queries_not_using_indexes OFF
log_slave_updates OFF
log_slow_queries OFF
log_warnings 1
long_query_time 10
low_priority_updates OFF
lower_case_file_system ON
lower_case_table_names 1
max_allowed_packet 1048576
max_binlog_cache_size 4294967295
max_binlog_size 1073741824
max_connect_errors 10
max_connections 250
max_delayed_threads 20
max_error_count 64
max_heap_table_size 16777216
max_insert_delayed_threads 20
max_join_size 4294967295
max_length_for_sort_data 1024
max_prepared_stmt_count 16382
max_relay_log_size 0
max_seeks_for_key 4294967295
max_sort_length 1024
max_sp_recursion_depth 0
max_tmp_tables 32
max_user_connections 0
max_write_lock_count 4294967295
multi_range_count 256
myisam_data_pointer_size 6
myisam_max_sort_file_size 107374182400
myisam_recover_options OFF
myisam_repair_threads 1
myisam_sort_buffer_size 31457280
myisam_stats_method nulls_unequal
named_pipe OFF
net_buffer_length 16384
net_read_timeout 30
net_retry_count 10
net_write_timeout 60
new OFF
old_passwords OFF
open_files_limit 2048
optimizer_prune_level 1
optimizer_search_depth 62
pid_file E:\Program Files\MySQL\MySQL Server 5.0_INNODB\Data\DSVR005889.pid
port 3306
preload_buffer_size 32768
profiling OFF
profiling_history_size 15
protocol_version 10
query_alloc_block_size 8192
query_cache_limit 1048576
query_cache_min_res_unit 4096
query_cache_size 87031808
query_cache_type ON
query_cache_wlock_invalidate OFF
query_prealloc_size 8192
range_alloc_block_size 2048
read_buffer_size 61440
read_only OFF
read_rnd_buffer_size 258048
relay_log_purge ON
relay_log_space_limit 0
rpl_recovery_rank 0
secure_auth OFF
secure_file_priv (null)
shared_memory OFF
shared_memory_base_name MYSQL
server_id 0
skip_external_locking ON
skip_networking OFF
skip_show_database OFF
slave_compressed_protocol OFF
slave_load_tmpdir e:\temp\
slave_net_timeout 3600
slave_skip_errors OFF
slave_transaction_retries 10
slow_launch_time 2
sort_buffer_size 1048568
sql_big_selects ON
sql_notes ON
sql_warnings OFF
ssl_ca (null)
ssl_capath (null)
ssl_cert (null)
ssl_cipher (null)
ssl_key (null)
storage_engine InnoDB
sync_binlog 0
sync_frm ON
system_time_zone GMT Standard Time
table_cache 894
table_lock_wait_timeout 50
table_type InnoDB
thread_cache_size 38
thread_stack 196608
time_format %H:%i:%s
time_zone SYSTEM
timed_mutexes OFF
tmp_table_size 104857600
tmpdir e:/temp
transaction_alloc_block_size 8192
transaction_prealloc_size 4096
tx_isolation REPEATABLE-READ
updatable_views_with_limit YES
version 5.0.45-community-nt
version_comment MySQL Community Edition (GPL)
version_compile_machine ia32
version_compile_os Win32
wait_timeout 28800
UmeshSenior Principal Technical Support EngineerCommented:

>>The machine is running IIS and MySQL 5 server. I am serving one website that does get quite a lot of traffic. Approx 300,000 pages (majority database driven).

I'll look into the details provided by you & ell you anything needed to be change.
BTW, I won't suggest you to run MySQL and IIS on same machine ..and lastly I never had a chance to hear from anyone about MySQL's performance on Windows platform...

Just a thought

innodb_buffer_pool_size=1000M ----> 1G

Normally when it's assumed that setting 60-80% total RAM to innodb_buffer_pool_size is good but you can't do it at the moment as again the questions comes if you allow more memory to it then what about OS and IIS???   also I see you have assigned 150M to key_buffer_size which is shouldn't be thr or should  be less if you are not going to use MyISAM at all.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
MySQL Server

From novice to tech pro — start learning today.