Link to home
Start Free TrialLog in
Avatar of Faith Victory
Faith VictoryFlag for United States of America

asked on

RAM Space issues on RDS Prod Instance.

We have a MySQL RDS instance that keeps running out of RAM space we have provisioned 16 GB to it but the weird part is innodb consume all 12GB thats its allocated and never releases anything.  maybe thats not weird, maybe its expected to be in cache like that to speed up queries and such, but, it still leaks beyond that, or something else is consuming it, etc.  Were obviously missing something. When the RAM hits 0 it crashes and dies. when we restart or cycle the RDS engine it always releases the memory as AWS reboots not only the service but the entire server that the MySQL engine is installed on. I also have historical process and memory dumps from prior exhaustion events. In those prior exhaustion events I was unable to uncover any memory consuming processes that were outside the acceptable limits imposed by the parameters set in the engineUser generated image

Avatar of slightwv (䄆 Netminder)
slightwv (䄆 Netminder)
As suggested in your previous question:
https://www.experts-exchange.com/questions/29244481/MySQL-Prod-Instance-running-out-of-RAM-Space.html

You should be able to see the process list to see what is consuming the memory.

Not sure MySQL in RDS is the same as Postres, but under monitoring, you can list the OS processes:
User generated image
Avatar of Faith Victory
Faith Victory
Flag of United States of America image

ASKER

event_namecurrent_countcurrent_alloccurrent_avg_allochigh_counthigh_allochigh_avg_alloc


















memory/innodb/buf_buf_pool8811.23 GiB130.69 MiB8811.23 GiB130.69 MiB


















memory/sql/sp_head::main_mem_root80337625.92 MiB7.98 KiB85630721.59 MiB8.63 KiB


















memory/innodb/hash0hash32306.62 MiB9.58 MiB32306.62 MiB9.58 MiB


















memory/sql/TABLE32032168.82 MiB5.40 KiB32580179.06 MiB5.63 KiB


















memory/innodb/memory1896564.07 MiB3.46 KiB40898373.10 MiB9.34 KiB


















memory/performance_schema/events_statements_summary_by_digest139.67 MiB39.67 MiB139.67 MiB39.67 MiB


















memory/performance_schema/table_handles436.25 MiB9.06 MiB436.25 MiB9.06 MiB


















memory/temptable/physical_ram2929.00 MiB1.00 MiB65651.00 MiB10.02 MiB


















memory/innodb/ut0link_buf224.00 MiB12.00 MiB224.00 MiB12.00 MiB


















memory/innodb/lock0lock1038122.22 MiB2.19 KiB1038122.22 MiB2.19 KiB


















memory/innodb/row0sel75921.12 MiB28.49 KiB94731.24 MiB33.78 KiB


















memory/mysys/KEY_CACHE316.00 MiB5.33 MiB316.00 MiB5.33 MiB


















memory/innodb/ha_innodb411013.91 MiB3.47 KiB433216.61 MiB3.93 KiB


















memory/performance_schema/events_statements_history_long113.89 MiB13.89 MiB113.89 MiB13.89 MiB


















memory/performance_schema/events_errors_summary_by_thread_by_error25711.93 MiB47.52 KiB25711.93 MiB47.52 KiB


















memory/performance_schema/events_statements_summary_by_thread_by_event_name19.79 MiB9.79 MiB19.79 MiB9.79 MiB


















memory/performance_schema/events_statements_summary_by_digest.digest_text19.77 MiB9.77 MiB19.77 MiB9.77 MiB


















memory/performance_schema/events_statements_history_long.digest_text19.77 MiB9.77 MiB19.77 MiB9.77 MiB


















memory/performance_schema/events_statements_history_long.sql_text19.77 MiB9.77 MiB19.77 MiB9.77 MiB


















memory/sql/NET::buff158.47 MiB578.14 KiB371596.07 MiB26.48 KiB


















memory/sql/TABLE_SHARE::mem_root21978.37 MiB3.90 KiB21978.43 MiB3.93 KiB


















memory/sql/dd::objects222397.94 MiB 374 bytes255258.88 MiB 364 bytes


















memory/performance_schema/memory_summary_by_thread_by_event_name17.91 MiB7.91 MiB17.91 MiB7.91 MiB


















memory/sql/String::value23736.97 MiB3.01 KiB27521195.80 MiB7.29 KiB


















memory/innodb/sync0arr26.25 MiB3.13 MiB26.25 MiB3.13 MiB


















memory/performance_schema/events_errors_summary_by_account_by_error1295.96 MiB47.33 KiB1295.96 MiB47.33 KiB


















memory/performance_schema/events_errors_summary_by_host_by_error1295.96 MiB47.33 KiB1295.96 MiB47.33 KiB


















memory/performance_schema/events_errors_summary_by_user_by_error1295.96 MiB47.33 KiB1295.96 MiB47.33 KiB


















memory/sql/log_sink_pfs15.00 MiB5.00 MiB25.02 MiB2.51 MiB


















memory/performance_schema/events_statements_summary_by_account_by_event_name14.90 MiB4.90 MiB14.90 MiB4.90 MiB


















memory/performance_schema/events_statements_summary_by_host_by_event_name14.90 MiB4.90 MiB14.90 MiB4.90 MiB


















memory/performance_schema/events_statements_summary_by_user_by_event_name14.90 MiB4.90 MiB14.90 MiB4.90 MiB


















memory/mysqld_openssl/openssl_malloc101554.52 MiB 466 bytes1957840014.80 GiB 811 bytes


















memory/performance_schema/events_waits_summary_by_thread_by_event_name14.33 MiB4.33 MiB14.33 MiB4.33 MiB


















memory/innodb/ut0pool14.00 MiB4.00 MiB14.00 MiB4.00 MiB


















memory/performance_schema/table_shares14.00 MiB4.00 MiB14.00 MiB4.00 MiB


















memory/performance_schema/memory_summary_by_account_by_event_name13.96 MiB3.96 MiB13.96 MiB3.96 MiB


















memory/performance_schema/memory_summary_by_host_by_event_name13.96 MiB3.96 MiB13.96 MiB3.96 MiB


















memory/performance_schema/memory_summary_by_user_by_event_name13.96 MiB3.96 MiB13.96 MiB3.96 MiB


















memory/performance_schema/events_statements_history13.55 MiB3.55 MiB13.55 MiB3.55 MiB


















memory/performance_schema/events_statements_current13.55 MiB3.55 MiB13.55 MiB3.55 MiB


















memory/performance_schema/events_transactions_history_long13.28 MiB3.28 MiB13.28 MiB3.28 MiB


















memory/mysys/IO_CACHE2862.97 MiB10.64 KiB4064.28 MiB10.80 KiB


















memory/innodb/ut0new239572.78 MiB 121 bytes243853.11 MiB 133 bytes


















memory/performance_schema/file_instances12.75 MiB2.75 MiB12.75 MiB2.75 MiB


















memory/performance_schema/table_io_waits_summary_by_index_usage12.75 MiB2.75 MiB12.75 MiB2.75 MiB


















memory/innodb/os0file1342.60 MiB19.90 KiB1342.60 MiB19.90 KiB


















memory/performance_schema/events_statements_history.digest_text12.50 MiB2.50 MiB12.50 MiB2.50 MiB


















memory/performance_schema/events_statements_history.sql_text12.50 MiB2.50 MiB12.50 MiB2.50 MiB


















memory/performance_schema/events_statements_current.digest_text12.50 MiB2.50 MiB12.50 MiB2.50 MiB


















memory/performance_schema/events_statements_current.sql_text12.50 MiB2.50 MiB12.50 MiB2.50 MiB


















memory/sql/THD::main_mem_root1492.41 MiB16.56 KiB160418864.91 MiB5.52 KiB


















memory/performance_schema/events_waits_summary_by_account_by_event_name12.16 MiB2.16 MiB12.16 MiB2.16 MiB


















memory/performance_schema/events_waits_summary_by_host_by_event_name12.16 MiB2.16 MiB12.16 MiB2.16 MiB


















memory/performance_schema/events_waits_summary_by_user_by_event_name12.16 MiB2.16 MiB12.16 MiB2.16 MiB


















memory/sql/dd::String_type258712.06 MiB  83 bytes290523.47 MiB 125 bytes


















memory/performance_schema/events_waits_history_long11.68 MiB1.68 MiB11.68 MiB1.68 MiB


















memory/performance_schema/prepared_statements_instances11.62 MiB1.62 MiB11.62 MiB1.62 MiB


















memory/innodb/trx0trx51901.43 MiB 288 bytes51901.43 MiB 288 bytes


















memory/performance_schema/events_stages_summary_by_thread_by_event_name11.37 MiB1.37 MiB11.37 MiB1.37 MiB


















memory/mysys/lf_node127001.36 MiB 112 bytes127001.36 MiB 112 bytes


















memory/performance_schema/table_lock_waits_summary_by_table11.34 MiB1.34 MiB11.34 MiB1.34 MiB


















memory/performance_schema/mutex_instances161.25 MiB80.00 KiB161.25 MiB80.00 KiB


















memory/performance_schema/threads11.05 MiB1.05 MiB11.05 MiB1.05 MiB


















memory/performance_schema/events_stages_history_long11015.62 KiB1015.62 KiB11015.62 KiB1015.62 KiB


















memory/performance_schema/events_transactions_history1860.00 KiB860.00 KiB1860.00 KiB860.00 KiB


















memory/performance_schema/events_stages_summary_by_account_by_event_name1700.00 KiB700.00 KiB1700.00 KiB700.00 KiB


















memory/performance_schema/events_stages_summary_by_host_by_event_name1700.00 KiB700.00 KiB1700.00 KiB700.00 KiB


















memory/performance_schema/events_stages_summary_by_user_by_event_name1700.00 KiB700.00 KiB1700.00 KiB700.00 KiB


















memory/mysys/lf_slist19526610.19 KiB  32 bytes19526610.19 KiB  32 bytes


















memory/performance_schema/metadata_locks1512.00 KiB512.00 KiB1512.00 KiB512.00 KiB


















memory/performance_schema/events_statements_summary_by_program1448.00 KiB448.00 KiB1448.00 KiB448.00 KiB


















memory/performance_schema/events_waits_history1440.00 KiB440.00 KiB1440.00 KiB440.00 KiB


















memory/sql/errmsgs::server1354.81 KiB354.81 KiB1354.81 KiB354.81 KiB


















memory/performance_schema/rwlock_instances4352.00 KiB88.00 KiB4352.00 KiB88.00 KiB


















memory/innodb/read0read1025288.28 KiB 288 bytes1025288.28 KiB 288 bytes


















memory/mysys/lf_dynarray92269.01 KiB2.92 KiB92269.01 KiB2.92 KiB


















memory/performance_schema/events_stages_history1260.00 KiB260.00 KiB1260.00 KiB260.00 KiB


















memory/performance_schema/file_handle1256.00 KiB256.00 KiB1256.00 KiB256.00 KiB


















memory/innodb/rem0rec4111213.69 KiB  53 bytes4400227.60 KiB  52 bytes


















memory/sql/plugin_mem_root67200.98 KiB3.00 KiB67200.98 KiB3.00 KiB


















memory/innodb/trx0i_s2200.17 KiB100.09 KiB2200.17 KiB100.09 KiB


















memory/innodb/trx0undo569195.59 KiB 352 bytes584200.75 KiB 352 bytes


















memory/sql/dd::infrastructure2039157.34 KiB  79 bytes12489020825.81 MiB  69 bytes


















memory/performance_schema/events_errors_summary_global_by_error1154.56 KiB154.56 KiB1154.56 KiB154.56 KiB


















memory/innodb/fil0fil372139.50 KiB 384 bytes372139.50 KiB 384 bytes


















memory/sql/Prepared_statement::main_mem_root11134.09 KiB12.19 KiB205832.12 MiB15.98 KiB


















memory/performance_schema/session_connect_attrs1128.00 KiB128.00 KiB1128.00 KiB128.00 KiB


















memory/performance_schema/accounts1112.00 KiB112.00 KiB1112.00 KiB112.00 KiB


















memory/performance_schema/hosts196.00 KiB96.00 KiB196.00 KiB96.00 KiB


















memory/innodb/trx0rseg38893.56 KiB 246 bytes38893.56 KiB 246 bytes


















memory/performance_schema/memory_class184.38 KiB84.38 KiB184.38 KiB84.38 KiB


















memory/performance_schema/users180.00 KiB80.00 KiB180.00 KiB80.00 KiB


















memory/performance_schema/socket_instances178.00 KiB78.00 KiB178.00 KiB78.00 KiB


















memory/performance_schema/mutex_class175.00 KiB75.00 KiB175.00 KiB75.00 KiB


















memory/innodb/buf0buf174.66 KiB74.66 KiB174.66 KiB74.66 KiB


















memory/innodb/mem0mem109664.41 KiB  60 bytes114766.85 KiB  59 bytes


















memory/performance_schema/setup_actors164.00 KiB64.00 KiB164.00 KiB64.00 KiB


















memory/innodb/trx0sys158.95 KiB58.95 KiB158.95 KiB58.95 KiB


















memory/performance_schema/setup_objects156.00 KiB56.00 KiB156.00 KiB56.00 KiB


















memory/performance_schema/cond_instances354.00 KiB18.00 KiB354.00 KiB18.00 KiB


















memory/innodb/btr0pcur97752.03 KiB  54 bytes102854.65 KiB  54 bytes


















memory/innodb/dict0dict33047.01 KiB 145 bytes33751.73 KiB 157 bytes


















memory/performance_schema/stage_class143.75 KiB43.75 KiB143.75 KiB43.75 KiB


















memory/sql/binlog_cache_mngr4742.96 KiB 936 bytes10595.98 KiB 936 bytes


















memory/performance_schema/statement_class140.88 KiB40.88 KiB140.88 KiB40.88 KiB


















memory/performance_schema/events_statements_summary_global_by_event_name139.17 KiB39.17 KiB139.17 KiB39.17 KiB


















memory/mysys/TREE338.02 KiB12.67 KiB1361.40 MiB10.52 KiB


















memory/sql/user_var_entry42835.93 KiB  85 bytes81467.46 KiB  84 bytes


















memory/innodb/log0log332.11 KiB10.70 KiB332.11 KiB10.70 KiB


















memory/sql/tz_storage132.01 KiB32.01 KiB132.01 KiB32.01 KiB


















memory/sql/Filesort_buffer::sort_keys132.00 KiB32.00 KiB275389.19 MiB1.42 MiB


















memory/performance_schema/memory_summary_global_by_event_name131.64 KiB31.64 KiB131.64 KiB31.64 KiB


















memory/innodb/ut_lock_free_hash_t26028.21 KiB 111 bytes26028.21 KiB 111 bytes


















memory/sql/THD::sp_cache40425.41 KiB  64 bytes78149.99 KiB  65 bytes


















memory/performance_schema/cond_class125.00 KiB25.00 KiB125.00 KiB25.00 KiB


















memory/performance_schema/thread_class125.00 KiB25.00 KiB125.00 KiB25.00 KiB


















memory/performance_schema/file_class125.00 KiB25.00 KiB125.00 KiB25.00 KiB


















memory/sql/LOG_POS_COORD4723.55 KiB 513 bytes6733.57 KiB 513 bytes


















memory/performance_schema/events_transactions_summary_by_thread_by_event_name122.00 KiB22.00 KiB122.00 KiB22.00 KiB


















memory/sql/THD::variables15219.00 KiB 128 bytes43654.50 KiB 128 bytes


















memory/innodb/lexyy316.17 KiB5.39 KiB316.17 KiB5.39 KiB


















memory/performance_schema/rwlock_class115.00 KiB15.00 KiB115.00 KiB15.00 KiB


















memory/sql/THD::Session_tracker23411.69 KiB  51 bytes480362.34 MiB  51 bytes


















memory/sql/thd_timer13611.69 KiB  88 bytes39734.12 KiB  88 bytes


















memory/innodb/sync0sharded_rw111.03 KiB11.03 KiB111.03 KiB11.03 KiB


















memory/performance_schema/events_transactions_summary_by_account_by_event_name111.00 KiB11.00 KiB111.00 KiB11.00 KiB


















memory/performance_schema/events_transactions_summary_by_host_by_event_name111.00 KiB11.00 KiB111.00 KiB11.00 KiB


















memory/performance_schema/events_transactions_summary_by_user_by_event_name111.00 KiB11.00 KiB111.00 KiB11.00 KiB


















memory/vio/vio1010.47 KiB1.05 KiB36873.77 MiB1.05 KiB


















memory/sql/sql_acl_mem69.53 KiB1.59 KiB1227.42 KiB2.29 KiB


















memory/sql/handlerton::objects107.89 KiB 808 bytes107.89 KiB 808 bytes


















memory/sql/user_var_entry::value1217.19 KiB  60 bytes28817.70 KiB  62 bytes


















memory/performance_schema/events_stages_summary_global_by_event_name15.47 KiB5.47 KiB15.47 KiB5.47 KiB


















memory/performance_schema/scalable_buffer414.11 KiB 102 bytes414.11 KiB 102 bytes


















memory/sql/THD::Session_sysvar_resource_manager543.53 KiB  67 bytes7412409.64 KiB  56 bytes


















memory/sql/Prepared_statement::infrastructure403.19 KiB  81 bytes72125.51 KiB  36 bytes


















memory/performance_schema/socket_class13.12 KiB3.12 KiB13.12 KiB3.12 KiB


















memory/sql/acl_cache512.77 KiB  55 bytes573.05 KiB  54 bytes


















memory/sql/THD::db1501.92 KiB  13 bytes1862.37 KiB  13 bytes


















memory/sql/host_cache::hostname271.79 KiB  67 bytes271.86 KiB  70 bytes


















memory/sql/sql_acl_memex81.55 KiB 198 bytes163.09 KiB 198 bytes


















memory/sql/User_level_lock181.55 KiB  88 bytes443.78 KiB  88 bytes


















memory/sql/THD::handler_tables_hash181.55 KiB  88 bytes443.78 KiB  88 bytes


















memory/sql/udf_mem91.48 KiB 168 bytes91.48 KiB 168 bytes


















memory/innodb/adaptive hash index111.35 KiB 125 bytes111.35 KiB 125 bytes


















memory/sql/ST_SCHEMA_TABLE281.31 KiB  48 bytes281.31 KiB  48 bytes


















memory/innodb/undo::Tablespaces31.16 KiB 394 bytes31.16 KiB 394 bytes


















memory/innodb/dict_stats_bg_recalc_pool_t21.09 KiB 560 bytes21.09 KiB 560 bytes


















memory/sql/plugin_bookmark171.09 KiB  65 bytes171.12 KiB  67 bytes


















memory/innodb/trx0purge6 958 bytes 159 bytes71.02 KiB 149 bytes


















memory/sql/TABLE_RULE_ENT17 902 bytes  53 bytes251.24 KiB  50 bytes


















memory/innodb/buf0flu2 784 bytes 392 bytes2 784 bytes 392 bytes


















memory/sql/errmsgs::handler1 720 bytes 720 bytes1 720 bytes 720 bytes


















memory/myisam/ft_stopwords1 696 bytes 696 bytes1 696 bytes 696 bytes


















memory/innodb/log and page archiver3 688 bytes 229 bytes4 841 bytes 210 bytes


















memory/innodb/srv0srv3 656 bytes 218 bytes3 656 bytes 218 bytes


















memory/sql/mysql_plugin1 512 bytes 512 bytes2 768 bytes 384 bytes


















memory/innodb/log0recv1 432 bytes 432 bytes22.00 MiB1.00 MiB


















memory/sql/TABLE::sort_io_cache1 312 bytes 312 bytes21465.20 KiB 312 bytes


















memory/mysys/my_file_info8 279 bytes  34 bytes24 697 bytes  29 bytes


















memory/sql/global_system_variables2 256 bytes 128 bytes3 320 bytes 106 bytes


















memory/innodb/clone1 256 bytes 256 bytes1 256 bytes 256 bytes


















memory/sql/Event_scheduler::scheduler_param1 240 bytes 240 bytes2 256 bytes 128 bytes


















memory/sql/Sid_map::Node4 236 bytes  59 bytes9 676 bytes  75 bytes


















memory/sql/Mutex_cond_array::Mutex_cond2 224 bytes 112 bytes2 224 bytes 112 bytes


















memory/sql/log_error::stack6 166 bytes  27 bytes147185.02 KiB1.26 KiB


















memory/innodb/buf_stat_per_index_t2 152 bytes  76 bytes2 152 bytes  76 bytes


















memory/mysys/my_err_head6 144 bytes  24 bytes6 144 bytes  24 bytes


















memory/innodb/ibuf0ibuf1 144 bytes 144 bytes1 144 bytes 144 bytes


















memory/sql/root1 136 bytes 136 bytes1 136 bytes 136 bytes


















memory/sql/Rpl_info_table6 102 bytes  17 bytes6 102 bytes  17 bytes


















memory/innodb/usr0sess2  96 bytes  48 bytes2  96 bytes  48 bytes


















memory/innodb/ut0wqueue1  96 bytes  96 bytes1  96 bytes  96 bytes


















memory/sql/Owned_gtids::sidno_to_hash1  88 bytes  88 bytes1  88 bytes  88 bytes


















memory/sql/db_worker_hash_entry1  88 bytes  88 bytes1  88 bytes  88 bytes


















memory/sql/user_conn1  88 bytes  88 bytes1  88 bytes  88 bytes


















memory/csv/TINA_SHARE1  88 bytes  88 bytes1  88 bytes  88 bytes


















memory/blackhole/blackhole_share1  88 bytes  88 bytes1  88 bytes  88 bytes


















memory/sql/servers_cache1  88 bytes  88 bytes1  88 bytes  88 bytes


















memory/sql/File_query_log::name2  86 bytes  43 bytes2  86 bytes  43 bytes


















memory/sql/mysql_plugin_dl1  82 bytes  82 bytes1  82 bytes  82 bytes


















memory/innodb/trx_sys_t::rw_trx_ids1  64 bytes  64 bytes2 112 bytes  56 bytes


















memory/innodb/pars0lex1  64 bytes  64 bytes1  64 bytes  64 bytes


















memory/sql/MYSQL_BIN_LOG::index1  48 bytes  48 bytes1  48 bytes  48 bytes


















memory/sql/MYSQL_LOCK1  48 bytes  48 bytes32318.05 KiB  57 bytes


















memory/innodb/ut0list1  48 bytes  48 bytes1  48 bytes  48 bytes


















memory/sql/MYSQL_LOG::name1  42 bytes  42 bytes2  84 bytes  42 bytes


















memory/sql/MYSQL_BIN_LOG::basename1  42 bytes  42 bytes1  42 bytes  42 bytes


















memory/sql/opt_bin_logname1  42 bytes  42 bytes1  42 bytes  42 bytes


















memory/mysys/MY_BITMAP::bitmap10  40 bytes   4 bytes5816.06 KiB  10 bytes


















memory/sql/MYSQL_RELAY_LOG::index1  39 bytes  39 bytes1  39 bytes  39 bytes


















memory/sql/MYSQL_RELAY_LOG::basename1  33 bytes  33 bytes1  33 bytes  33 bytes


















memory/mysys/MY_TMPDIR::full_list2  23 bytes  11 bytes2  23 bytes  11 bytes


















memory/sql/NAMED_ILINK::name1   8 bytes   8 bytes1   8 bytes   8 bytes


















memory/vio/ssl_fd1   8 bytes   8 bytes1   8 bytes   8 bytes


















Avatar of Faith Victory
Faith Victory
Flag of United States of America image

ASKER

I don't really see what's cosuming so much data from this OS process list
ASKER CERTIFIED SOLUTION
Avatar of slightwv (䄆 Netminder)
slightwv (䄆 Netminder)
Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
SOLUTION
Avatar of Adelaido Jimenez
Adelaido Jimenez
Flag of United States of America image
Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
SOLUTION
Avatar of Ramasamy P
Ramasamy P
Flag of India image
Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial