JP_TechGroup
asked on
MySQL Server 5.5.36 Tuning
I recently inherited a MySQL dbase that is hosting data for an Access Front end using ODBC. In essence the Schema consists of about 8 large tables (6 Million rows +) and an number of secondary tables. The data is stored thus:
Let's say I had a record that held name, City, State, Zip
One would expect a single row with each piece in a column formatted to hold that data, indexed with a unique primary key.
Using that same model, this database assigns an index to the type of data being stored (1=name, 2=City,3=address,4=state) and an index for the root context of the dataset (let's say its the name)
Then stores each piece of data on its own row, referencing the data type index and the root index. Each row has a column to hold different data types (Double, Date/Time, Varchar, etc)
So in order to retrieve the complete record, a query must join the table to itself 4 times on the root context id and then pull each column.
With this schema, anything one does is going to be slow, of course. I have been tasked with tuning the database as best as possible to address the ever increasing lag time on queries.
I have plenty of resources available. CPU usages stays below 25% and memory below 40%. Attcahed is the my.cnf.
I don't even know where to start on this. Discuss and thank you!
Let's say I had a record that held name, City, State, Zip
One would expect a single row with each piece in a column formatted to hold that data, indexed with a unique primary key.
Using that same model, this database assigns an index to the type of data being stored (1=name, 2=City,3=address,4=state) and an index for the root context of the dataset (let's say its the name)
Then stores each piece of data on its own row, referencing the data type index and the root index. Each row has a column to hold different data types (Double, Date/Time, Varchar, etc)
So in order to retrieve the complete record, a query must join the table to itself 4 times on the root context id and then pull each column.
With this schema, anything one does is going to be slow, of course. I have been tasked with tuning the database as best as possible to address the ever increasing lag time on queries.
I have plenty of resources available. CPU usages stays below 25% and memory below 40%. Attcahed is the my.cnf.
I don't even know where to start on this. Discuss and thank you!
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
First I'd find out how many different places the current queries exist. So you would know how many places you would have to update the queries after turning it into a 'normal' database that doesn't require 4 joins to get a single record.
ASKER
Thanks for the feedback guys. I have already run a tuning script and did not get much that was helpful.
I have enabled slow query logging, but I am reasonably sure that most queries here are going to be slow!
Re-writing the queries so they will be faster id not really an option. How? My schema structure is what it is. Altering the database and the front end to change that is an enormous re-write of a project I took ownership of yesterday!
I'm screwed, aren't I?
I have enabled slow query logging, but I am reasonably sure that most queries here are going to be slow!
Re-writing the queries so they will be faster id not really an option. How? My schema structure is what it is. Altering the database and the front end to change that is an enormous re-write of a project I took ownership of yesterday!
I'm screwed, aren't I?
slow queries is a programming error. Can you post a query or two with respective DDL?
16GB to hold full database in RAM is not expensive.
You will be wasting lots of CPU
16GB to hold full database in RAM is not expensive.
You will be wasting lots of CPU
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Thanks Dave. That was the conclusion I had come to as well. You hope that you don't know what you're talking about and throw it out there...
You're welcome. I had a table like that... but at a couple of hundred rows performance wasn't a problem.
Lets attack where it hurts most with minimal rewrite of code:
Easiest:
select a.x where a.y in (whatever list) - obviously simplest index on y will make things roll.
If there is an index and full read happens try to sniff around if DB server ever/often crashed. ndexes might be invalidated over time, and reindexing would get them back to work.
Say in the case no query gets prepared you might want to add parentheses (or optimize ANDs and ORs to make less (ISAM) teable reads,)in select where or reorder _ JOIN to save time on query planner/optimizer running for each query. Usually you run EXPLAIN with query and try to re-arrange it to match optimizer result.
Other thing to search in slow query log : select a.* 0> it is wrong, listing fields in query is faster
SELECT queries dont change data, you can play around with them all the time.
Easiest:
select a.x where a.y in (whatever list) - obviously simplest index on y will make things roll.
If there is an index and full read happens try to sniff around if DB server ever/often crashed. ndexes might be invalidated over time, and reindexing would get them back to work.
Say in the case no query gets prepared you might want to add parentheses (or optimize ANDs and ORs to make less (ISAM) teable reads,)in select where or reorder _ JOIN to save time on query planner/optimizer running for each query. Usually you run EXPLAIN with query and try to re-arrange it to match optimizer result.
Other thing to search in slow query log : select a.* 0> it is wrong, listing fields in query is faster
SELECT queries dont change data, you can play around with them all the time.
ASKER
GLOBAL STATUS:
Aborted_clients...........
Aborted_connects..........
Binlog_cache_disk_use.....
Binlog_cache_use..........
Binlog_stmt_cache_disk_use
Binlog_stmt_cache_use.....
Bytes_received............
Bytes_sent................
Com_admin_commands........
Com_assign_to_keycache....
Com_alter_db..............
Com_alter_db_upgrade......
Com_alter_event...........
Com_alter_function........
Com_alter_procedure.......
Com_alter_server..........
Com_alter_table...........
Com_alter_tablespace......
Com_analyze...............
Com_begin.................
Com_binlog................
Com_call_procedure........
Com_change_db.............
Com_change_master.........
Com_check.................
Com_checksum..............
Com_commit................
Com_create_db.............
Com_create_event..........
Com_create_function.......
Com_create_index..........
Com_create_procedure......
Com_create_server.........
Com_create_table..........
Com_create_trigger........
Com_create_udf............
Com_create_user...........
Com_create_view...........
Com_dealloc_sql...........
Com_delete................
Com_delete_multi..........
Com_do....................
Com_drop_db...............
Com_drop_event............
Com_drop_function.........
Com_drop_index............
Com_drop_procedure........
Com_drop_server...........
Com_drop_table............
Com_drop_trigger..........
Com_drop_user.............
Com_drop_view.............
Com_empty_query...........
Com_execute_sql...........
Com_flush.................
Com_grant.................
Com_ha_close..............
Com_ha_open...............
Com_ha_read...............
Com_help..................
Com_insert................
Com_insert_select.........
Com_install_plugin........
Com_kill..................
Com_load..................
Com_lock_tables...........
Com_optimize..............
Com_preload_keys..........
Com_prepare_sql...........
Com_purge.................
Com_purge_before_date.....
Com_release_savepoint.....
Com_rename_table..........
Com_rename_user...........
Com_repair................
Com_replace...............
Com_replace_select........
Com_reset.................
Com_resignal..............
Com_revoke................
Com_revoke_all............
Com_rollback..............
Com_rollback_to_savepoint.
Com_savepoint.............
Com_select................
Com_set_option............
Com_signal................
Com_show_authors..........
Com_show_binlog_events....
Com_show_binlogs..........
Com_show_charsets.........
Com_show_collations.......
Com_show_contributors.....
Com_show_create_db........
Com_show_create_event.....
Com_show_create_func......
Com_show_create_proc......
Com_show_create_table.....
Com_show_create_trigger...
Com_show_databases........
Com_show_engine_logs......
Com_show_engine_mutex.....
Com_show_engine_status....
Com_show_events...........
Com_show_errors...........
Com_show_fields...........
Com_show_function_status..
Com_show_grants...........
Com_show_keys.............
Com_show_master_status....
Com_show_open_tables......
Com_show_plugins..........
Com_show_privileges.......
Com_show_procedure_status.
Com_show_processlist......
Com_show_profile..........
Com_show_profiles.........
Com_show_relaylog_events..
Com_show_slave_hosts......
Com_show_slave_status.....
Com_show_status...........
Com_show_storage_engines..
Com_show_table_status.....
Com_show_tables...........
Com_show_triggers.........
Com_show_variables........
Com_show_warnings.........
Com_slave_start...........
Com_slave_stop............
Com_stmt_close............
Com_stmt_execute..........
Com_stmt_fetch............
Com_stmt_prepare..........
Com_stmt_reprepare........
Com_stmt_reset............
Com_stmt_send_long_data...
Com_truncate..............
Com_uninstall_plugin......
Com_unlock_tables.........
Com_update................
Com_update_multi..........
Com_xa_commit.............
Com_xa_end................
Com_xa_prepare............
Com_xa_recover............
Com_xa_rollback...........
Com_xa_start..............
Compression...............
Connections...............
Created_tmp_disk_tables...
Created_tmp_files.........
Created_tmp_tables........
Delayed_errors............
Delayed_insert_threads....
Delayed_writes............
Flush_commands............
Handler_commit............
Handler_delete............
Handler_discover..........
Handler_prepare...........
Handler_read_first........
Handler_read_key..........
Handler_read_last.........
Handler_read_next.........
Handler_read_prev.........
Handler_read_rnd..........
Handler_read_rnd_next.....
Handler_rollback..........
Handler_savepoint.........
Handler_savepoint_rollback
Handler_update............
Handler_write.............
Innodb_buffer_pool_pages_d
Innodb_buffer_pool_bytes_d
Innodb_buffer_pool_pages_d
Innodb_buffer_pool_bytes_d
Innodb_buffer_pool_pages_f
Innodb_buffer_pool_pages_f
Innodb_buffer_pool_pages_m
Innodb_buffer_pool_pages_t
Innodb_buffer_pool_read_ah
Innodb_buffer_pool_read_ah
Innodb_buffer_pool_read_ah
Innodb_buffer_pool_read_re
Innodb_buffer_pool_reads..
Innodb_buffer_pool_wait_fr
Innodb_buffer_pool_write_r
Innodb_data_fsyncs........
Innodb_data_pending_fsyncs
Innodb_data_pending_reads.
Innodb_data_pending_writes
Innodb_data_read..........
Innodb_data_reads.........
Innodb_data_writes........
Innodb_data_written.......
Innodb_dblwr_pages_written
Innodb_dblwr_writes.......
Innodb_have_atomic_builtin
Innodb_log_waits..........
Innodb_log_write_requests.
Innodb_log_writes.........
Innodb_os_log_fsyncs......
Innodb_os_log_pending_fsyn
Innodb_os_log_pending_writ
Innodb_os_log_written.....
Innodb_page_size..........
Innodb_pages_created......
Innodb_pages_read.........
Innodb_pages_written......
Innodb_row_lock_current_wa
Innodb_row_lock_time......
Innodb_row_lock_time_avg..
Innodb_row_lock_time_max..
Innodb_row_lock_waits.....
Innodb_rows_deleted.......
Innodb_rows_inserted......
Innodb_rows_read..........
Innodb_rows_updated.......
Innodb_truncated_status_wr
Key_blocks_not_flushed....
Key_blocks_unused.........
Key_blocks_used...........
Key_read_requests.........
Key_reads.................
Key_write_requests........
Key_writes................
Last_query_cost...........
Max_used_connections......
Not_flushed_delayed_rows..
Open_files................
Open_streams..............
Open_table_definitions....
Open_tables...............
Opened_files..............
Opened_table_definitions..
Opened_tables.............
Performance_schema_cond_cl
Performance_schema_cond_in
Performance_schema_file_cl
Performance_schema_file_ha
Performance_schema_file_in
Performance_schema_locker_
Performance_schema_mutex_c
Performance_schema_mutex_i
Performance_schema_rwlock_
Performance_schema_rwlock_
Performance_schema_table_h
Performance_schema_table_i
Performance_schema_thread_
Performance_schema_thread_
Prepared_stmt_count.......
Qcache_free_blocks........
Qcache_free_memory........
Qcache_hits...............
Qcache_inserts............
Qcache_lowmem_prunes......
Qcache_not_cached.........
Qcache_queries_in_cache...
Qcache_total_blocks.......
Queries...................
Questions.................
Rpl_status................
Select_full_join..........
Select_full_range_join....
Select_range..............
Select_range_check........
Select_scan...............
Slave_heartbeat_period....
Slave_open_temp_tables....
Slave_received_heartbeats.
Slave_retried_transactions
Slave_running.............
Slow_launch_threads.......
Slow_queries..............
Sort_merge_passes.........
Sort_range................
Sort_rows.................
Sort_scan.................
Ssl_accept_renegotiates...
Ssl_accepts...............
Ssl_callback_cache_hits...
Ssl_cipher................
Ssl_cipher_list...........
Ssl_client_connects.......
Ssl_connect_renegotiates..
Ssl_ctx_verify_depth......
Ssl_ctx_verify_mode.......
Ssl_default_timeout.......
Ssl_finished_accepts......
Ssl_finished_connects.....
Ssl_session_cache_hits....
Ssl_session_cache_misses..
Ssl_session_cache_mode....
Ssl_session_cache_overflow
Ssl_session_cache_size....
Ssl_session_cache_timeouts
Ssl_sessions_reused.......
Ssl_used_session_cache_ent
Ssl_verify_depth..........
Ssl_verify_mode...........
Ssl_version...............
Table_locks_immediate.....
Table_locks_waited........
Tc_log_max_pages_used.....
Tc_log_page_size..........
Tc_log_page_waits.........
Threads_cached............
Threads_connected.........
Threads_created...........
Threads_running...........
Uptime....................
Uptime_since_flush_status.
GLOBAL VARIABLES:
auto_increment_increment..
auto_increment_offset.....
autocommit................
automatic_sp_privileges...
back_log..................
basedir...................
big_tables................
binlog_cache_size.........
binlog_direct_non_transact
binlog_format.............
binlog_stmt_cache_size....
bulk_insert_buffer_size...
character_set_client......
character_set_connection..
character_set_database....
character_set_filesystem..
character_set_results.....
character_set_server......
character_set_system......
character_sets_dir........
collation_connection......
collation_database........
collation_server..........
completion_type...........
concurrent_insert.........
connect_timeout...........
datadir...................
date_format...............
datetime_format...........
default_storage_engine....
default_week_format.......
delay_key_write...........
delayed_insert_limit......
delayed_insert_timeout....
delayed_queue_size........
div_precision_increment...
engine_condition_pushdown.
event_scheduler...........
expire_logs_days..........
flush.....................
flush_time................
foreign_key_checks........
ft_boolean_syntax.........
ft_max_word_len...........
ft_min_word_len...........
ft_query_expansion_limit..
ft_stopword_file..........
general_log...............
general_log_file..........
group_concat_max_len......
have_compress.............
have_crypt................
have_csv..................
have_dynamic_loading......
have_geometry.............
have_innodb...............
have_ndbcluster...........
have_openssl..............
have_partitioning.........
have_profiling............
have_query_cache..........
have_rtree_keys...........
have_ssl..................
have_symlink..............
hostname..................
ignore_builtin_innodb.....
init_connect..............
init_file.................
init_slave................
innodb_adaptive_flushing..
innodb_adaptive_hash_index
innodb_additional_mem_pool
innodb_autoextend_incremen
innodb_autoinc_lock_mode..
innodb_buffer_pool_instanc
innodb_buffer_pool_size...
innodb_change_buffering...
innodb_checksums..........
innodb_commit_concurrency.
innodb_concurrency_tickets
innodb_data_file_path.....
innodb_data_home_dir......
innodb_doublewrite........
innodb_fast_shutdown......
innodb_file_format........
innodb_file_format_check..
innodb_file_format_max....
innodb_file_per_table.....
innodb_flush_log_at_trx_co
innodb_flush_method.......
innodb_force_load_corrupte
innodb_force_recovery.....
innodb_io_capacity........
innodb_large_prefix.......
innodb_lock_wait_timeout..
innodb_locks_unsafe_for_bi
innodb_log_buffer_size....
innodb_log_file_size......
innodb_log_files_in_group.
innodb_log_group_home_dir.
innodb_max_dirty_pages_pct
innodb_max_purge_lag......
innodb_mirrored_log_groups
innodb_old_blocks_pct.....
innodb_old_blocks_time....
innodb_open_files.........
innodb_print_all_deadlocks
innodb_purge_batch_size...
innodb_purge_threads......
innodb_random_read_ahead..
innodb_read_ahead_threshol
innodb_read_io_threads....
innodb_replication_delay..
innodb_rollback_on_timeout
innodb_rollback_segments..
innodb_spin_wait_delay....
innodb_stats_method.......
innodb_stats_on_metadata..
innodb_stats_sample_pages.
innodb_strict_mode........
innodb_support_xa.........
innodb_sync_spin_loops....
innodb_table_locks........
innodb_thread_concurrency.
innodb_thread_sleep_delay.
innodb_use_native_aio.....
innodb_use_sys_malloc.....
innodb_version............
innodb_write_io_threads...
interactive_timeout.......
join_buffer_size..........
keep_files_on_create......
key_buffer_size...........
key_cache_age_threshold...
key_cache_block_size......
key_cache_division_limit..
large_files_support.......
large_page_size...........
large_pages...............
lc_messages...............
lc_messages_dir...........
lc_time_names.............
license...................
local_infile..............
lock_wait_timeout.........
log.......................
log_bin...................
log_bin_trust_function_cre
log_error.................
log_output................
log_queries_not_using_inde
log_slave_updates.........
log_slow_queries..........
log_warnings..............
long_query_time...........
low_priority_updates......
lower_case_file_system....
lower_case_table_names....
max_allowed_packet........
max_binlog_cache_size.....
max_binlog_size...........
max_binlog_stmt_cache_size
max_connect_errors........
max_connections...........
max_delayed_threads.......
max_error_count...........
max_heap_table_size.......
max_insert_delayed_threads
max_join_size.............
max_length_for_sort_data..
max_long_data_size........
max_prepared_stmt_count...
max_relay_log_size........
max_seeks_for_key.........
max_sort_length...........
max_sp_recursion_depth....
max_tmp_tables............
max_user_connections......
max_write_lock_count......
metadata_locks_cache_size.
min_examined_row_limit....
multi_range_count.........
myisam_data_pointer_size..
myisam_max_sort_file_size.
myisam_mmap_size..........
myisam_recover_options....
myisam_repair_threads.....
myisam_sort_buffer_size...
myisam_stats_method.......
myisam_use_mmap...........
named_pipe................
net_buffer_length.........
net_read_timeout..........
net_retry_count...........
net_write_timeout.........
new.......................
old.......................
old_alter_table...........
old_passwords.............
open_files_limit..........
optimizer_prune_level.....
optimizer_search_depth....
optimizer_switch..........
performance_schema........
performance_schema_events_
performance_schema_events_
performance_schema_max_con
performance_schema_max_con
performance_schema_max_fil
performance_schema_max_fil
performance_schema_max_fil
performance_schema_max_mut
performance_schema_max_mut
performance_schema_max_rwl
performance_schema_max_rwl
performance_schema_max_tab
performance_schema_max_tab
performance_schema_max_thr
performance_schema_max_thr
pid_file..................
plugin_dir................
port......................
preload_buffer_size.......
profiling.................
profiling_history_size....
protocol_version..........
query_alloc_block_size....
query_cache_limit.........
query_cache_min_res_unit..
query_cache_size..........
query_cache_type..........
query_cache_wlock_invalida
query_prealloc_size.......
range_alloc_block_size....
read_buffer_size..........
read_only.................
read_rnd_buffer_size......
relay_log.................
relay_log_index...........
relay_log_info_file.......
relay_log_purge...........
relay_log_recovery........
relay_log_space_limit.....
report_host...............
report_password...........
report_port...............
report_user...............
rpl_recovery_rank.........
secure_auth...............
secure_file_priv..........
server_id.................
shared_memory.............
shared_memory_base_name...
skip_external_locking.....
skip_name_resolve.........
skip_networking...........
skip_show_database........
slave_compressed_protocol.
slave_exec_mode...........
slave_load_tmpdir.........
slave_max_allowed_packet..
slave_net_timeout.........
slave_skip_errors.........
slave_transaction_retries.
slave_type_conversions....
slow_launch_time..........
slow_query_log............
slow_query_log_file.......
socket....................
sort_buffer_size..........
sql_auto_is_null..........
sql_big_selects...........
sql_big_tables............
sql_buffer_result.........
sql_log_bin...............
sql_log_off...............
sql_low_priority_updates..
sql_max_join_size.........
sql_mode..................
sql_notes.................
sql_quote_show_create.....
sql_safe_updates..........
sql_select_limit..........
sql_slave_skip_counter....
sql_warnings..............
ssl_ca....................
ssl_capath................
ssl_cert..................
ssl_cipher................
ssl_key...................
storage_engine............
stored_program_cache......
sync_binlog...............
sync_frm..................
sync_master_info..........
sync_relay_log............
sync_relay_log_info.......
system_time_zone..........
table_definition_cache....
table_open_cache..........
thread_cache_size.........
thread_concurrency........
thread_handling...........
thread_stack..............
time_format...............
time_zone.................
timed_mutexes.............
tmp_table_size............
tmpdir....................
transaction_alloc_block_si
transaction_prealloc_size.
tx_isolation..............
unique_checks.............
updatable_views_with_limit
version...................
version_comment...........
version_compile_machine...
version_compile_os........
wait_timeout..............