How to solve a timeout problem in MariadB
I had my 10.2.15-MariaDB server crashed after a select count(*) from a 18.6 Gb table.
It's not a HeidiSQL error because it also happens if i do it using MariaDB's command prompt.
After this crash, i can't do a select count(*) over 60 seconds that i get the same error again.
It doesn't seem like a configuration issue as this weren't happening before the first crash.
I used to run several hours querys without a problem.
My interactive_timeout variable is set to 28800:
"Variable_name";"Value" "connect_timeout";"36000" "delayed_insert_timeout";"
I still have 62 Gb free in my hard drive (SSD) and my PC has 32 Gb. of RAM memory.
Thanks!
It's not a HeidiSQL error because it also happens if i do it using MariaDB's command prompt.
After this crash, i can't do a select count(*) over 60 seconds that i get the same error again.
It doesn't seem like a configuration issue as this weren't happening before the first crash.
I used to run several hours querys without a problem.
My interactive_timeout variable is set to 28800:
"Variable_name";"Value" "connect_timeout";"36000" "delayed_insert_timeout";"
I still have 62 Gb free in my hard drive (SSD) and my PC has 32 Gb. of RAM memory.
Thanks!
ASKER
I´m not a MariadB or database expert.
So, I have some difficulties to figure out what´s going wrong.
My error log from MariadB
So, I have some difficulties to figure out what´s going wrong.
My error log from MariadB
Server version: 10.2.15-MariaDB
key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=1
max_threads=65537
thread_count=7
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 136060 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
mysqld.exe!my_sigabrt_handler()[my_thr_init.c:487]
mysqld.exe!raise()[signal.cpp:516]
mysqld.exe!abort()[abort.cpp:71]
mysqld.exe!os_file_handle_error_cond_exit()[os0file.cc:5227]
mysqld.exe!os_aio_windows_handler()[os0file.cc:6574]
mysqld.exe!os_aio_handler()[os0file.cc:5701]
mysqld.exe!fil_aio_wait()[fil0fil.cc:5350]
mysqld.exe!io_handler_thread()[srv0start.cc:338]
KERNEL32.DLL!BaseThreadInitThunk()
ntdll.dll!RtlUserThreadStart()
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
2019-03-01 22:36:44 12560 [Note] InnoDB: Mutexes and rw_locks use Windows interlocked functions
2019-03-01 22:36:44 12560 [Note] InnoDB: Uses event mutexes
2019-03-01 22:36:44 12560 [Note] InnoDB: Compressed tables use zlib 1.2.3
2019-03-01 22:36:44 12560 [Note] InnoDB: Number of pools: 1
2019-03-01 22:36:44 12560 [Note] InnoDB: Using SSE2 crc32 instructions
2019-03-01 22:36:44 12560 [Note] InnoDB: Initializing buffer pool, total size = 8G, instances = 8, chunk size = 128M
2019-03-01 22:36:44 12560 [Note] InnoDB: Completed initialization of buffer pool
2019-03-01 22:36:45 12560 [Note] InnoDB: Highest supported file format is Barracuda.
2019-03-01 22:36:45 12560 [Note] InnoDB: Starting crash recovery from checkpoint LSN=1281846889734
2019-03-01 22:36:45 12560 [Note] InnoDB: 128 out of 128 rollback segments are active.
2019-03-01 22:36:45 12560 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2019-03-01 22:36:45 12560 [Note] InnoDB: Creating shared tablespace for temporary tables
2019-03-01 22:36:45 12560 [Note] InnoDB: Setting file '.\ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2019-03-01 22:36:45 12560 [Note] InnoDB: File '.\ibtmp1' size is now 12 MB.
2019-03-01 22:36:45 12560 [Note] InnoDB: Waiting for purge to start
2019-03-01 22:36:45 12560 [Note] InnoDB: 5.7.22 started; log sequence number 1281846889743
2019-03-01 22:36:45 5216 [Note] InnoDB: Loading buffer pool(s) from C:\Program Files\MariaDB 10.2\data\ib_buffer_pool
2019-03-01 22:36:45 12560 [Note] Plugin 'FEEDBACK' is disabled.
2019-03-01 22:36:45 12560 [Note] Reading of all Master_info entries succeded
2019-03-01 22:36:45 12560 [Note] Added new Master_info '' to hash table
2019-03-01 22:36:45 12560 [Note] C:\Program Files\MariaDB 10.2\bin\mysqld.exe: ready for connections.
Version: '10.2.15-MariaDB' socket: '' port: 0 mariadb.org binary distribution
2019-03-01 22:37:15 5216 [Note] InnoDB: Buffer pool(s) load completed at 190301 22:37:15
2019-03-02 0:29:19 3528 [ERROR] InnoDB: Operating system error number 23 in a file operation.
2019-03-02 0:29:19 3528 [Note] InnoDB: Some operating system error numbers are described at https://mariadb.com/kb/en/library/operating-system-error-codes/
2019-03-02 0:29:19 3528 [ERROR] InnoDB: File .\teste\dados_2013.ibd: 'Windows aio' returned OS error 223. Cannot continue operation
190302 0:29:19 [ERROR] mysqld got exception 0x80000003 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Server version: 10.2.15-MariaDB
key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=1
max_threads=65537
thread_count=7
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 136060 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
2019-03-02 0:29:19 12956 [ERROR] InnoDB: Operating system error number 23 in a file operation.
2019-03-02 0:29:19 12956 [Note] InnoDB: Some operating system error numbers are described at https://mariadb.com/kb/en/library/operating-system-error-codes/
2019-03-02 0:29:19 12956 [ERROR] InnoDB: File .\teste\dados_2013.ibd: 'Windows aio' returned OS error 223. Cannot continue operation
2019-03-02 0:48:42 9648 [Note] InnoDB: Mutexes and rw_locks use Windows interlocked functions
2019-03-02 0:48:42 9648 [Note] InnoDB: Uses event mutexes
2019-03-02 0:48:42 9648 [Note] InnoDB: Compressed tables use zlib 1.2.3
2019-03-02 0:48:42 9648 [Note] InnoDB: Number of pools: 1
2019-03-02 0:48:42 9648 [Note] InnoDB: Using SSE2 crc32 instructions
2019-03-02 0:48:42 9648 [Note] InnoDB: Initializing buffer pool, total size = 8G, instances = 8, chunk size = 128M
2019-03-02 0:48:42 9648 [Note] InnoDB: Completed initialization of buffer pool
2019-03-02 0:48:42 9648 [Note] InnoDB: Highest supported file format is Barracuda.
2019-03-02 0:48:42 9648 [Note] InnoDB: Starting crash recovery from checkpoint LSN=1281846889771
2019-03-02 0:48:43 9648 [Note] InnoDB: 128 out of 128 rollback segments are active.
2019-03-02 0:48:43 9648 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2019-03-02 0:48:43 9648 [Note] InnoDB: Creating shared tablespace for temporary tables
2019-03-02 0:48:43 9648 [Note] InnoDB: Setting file '.\ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2019-03-02 0:48:43 9648 [Note] InnoDB: File '.\ibtmp1' size is now 12 MB.
2019-03-02 0:48:43 9648 [Note] InnoDB: Waiting for purge to start
2019-03-02 0:48:43 9648 [Note] InnoDB: 5.7.22 started; log sequence number 1281846889780
2019-03-02 0:48:43 12924 [Note] InnoDB: Loading buffer pool(s) from C:\Program Files\MariaDB 10.2\data\ib_buffer_pool
2019-03-02 0:48:43 9648 [Note] Plugin 'FEEDBACK' is disabled.
2019-03-02 0:48:43 9648 [Note] Reading of all Master_info entries succeded
2019-03-02 0:48:43 9648 [Note] Added new Master_info '' to hash table
2019-03-02 0:48:43 9648 [Note] C:\Program Files\MariaDB 10.2\bin\mysqld.exe: ready for connections.
Version: '10.2.15-MariaDB' socket: '' port: 0 mariadb.org binary distribution
2019-03-02 0:49:10 12924 [Note] InnoDB: Buffer pool(s) load completed at 190302 0:49:10
forget the memory issue. you may still run into some with your settings but that was not the cause of the crash
the interesting information is here
2019-03-02 0:29:19 3528 InnoDB: Operating system error number 23 in a file operation.
2019-03-02 0:29:19 3528 [Note] InnoDB: Some operating system error numbers are described at https://mariadb.com/kb/en/library/operating-system-error-codes/
2019-03-02 0:29:19 3528 InnoDB: File .\teste\dados_2013.ibd: 'Windows aio' returned OS error 223. Cannot continue operation
190302 0:29:19 mysqld got exception 0x80000003 ;
i would assume you are using file_per_table and dados_2013 is a table in teste database which would be corrupt.
can you confirm that table is the one involved ?
the interesting information is here
2019-03-02 0:29:19 3528 InnoDB: Operating system error number 23 in a file operation.
2019-03-02 0:29:19 3528 [Note] InnoDB: Some operating system error numbers are described at https://mariadb.com/kb/en/library/operating-system-error-codes/
2019-03-02 0:29:19 3528 InnoDB: File .\teste\dados_2013.ibd: 'Windows aio' returned OS error 223. Cannot continue operation
190302 0:29:19 mysqld got exception 0x80000003 ;
i would assume you are using file_per_table and dados_2013 is a table in teste database which would be corrupt.
can you confirm that table is the one involved ?
i'm unsure but i guess you are hitting
ERROR_FILE_TOO_LARGE
223 (0xDF)
The file size exceeds the limit allowed and cannot be saved.
described here https://docs.microsoft.com/en-us/windows/desktop/debug/system-error-codes--0-499-
which seems strange because ntfs should allow for HUGE sizes.
can you check how big the file is and if it can be opened in whatever editor will allow to open very big files ?
ERROR_FILE_TOO_LARGE
223 (0xDF)
The file size exceeds the limit allowed and cannot be saved.
described here https://docs.microsoft.com/en-us/windows/desktop/debug/system-error-codes--0-499-
which seems strange because ntfs should allow for HUGE sizes.
can you check how big the file is and if it can be opened in whatever editor will allow to open very big files ?
ASKER
Yes, the table dados_2013 was involved in the first crash and after that Maria crash in other simple querys with another tables. Sometimes not
But dados_2013 was involved in the first crash after a simple : select count (*) from dados_2013.
But dados_2013 was involved in the first crash after a simple : select count (*) from dados_2013.
ASKER
The table dados_2013 has 20 Gb. I could open it in UltraEdit, but the free trial has expired. So, it´s a test that I can not run now.
PS1: After crash I have to restart MySQL process in the task manager to reconnect with MariaDb.
PS2: I can drop dados_2013 from dB. I have a backup.
PS1: After crash I have to restart MySQL process in the task manager to reconnect with MariaDb.
PS2: I can drop dados_2013 from dB. I have a backup.
whatever the running query, you can check whether the filename that appears in the logs is always the same. this should help determine between a problem with this specific file or you're hitting filesystem corruption / disk problems.
you have not answered regarding the filesize. you may or may not be reaching a well-known limit.
is your system 32 bits ? or possibly your mariadb version ?
ultraedit has tons of free alternatives. i'm pretty sure you can figure that out. you can also try to zip the file which will force the archieving program to read the whole file and complain in case there is corruption. obviously try while mariadb is not running.
regards
you have not answered regarding the filesize. you may or may not be reaching a well-known limit.
is your system 32 bits ? or possibly your mariadb version ?
ultraedit has tons of free alternatives. i'm pretty sure you can figure that out. you can also try to zip the file which will force the archieving program to read the whole file and complain in case there is corruption. obviously try while mariadb is not running.
regards
ASKER
The DB files are corrupted due to disk problem.
HD crashed. Solved.
Thanks a lot.
HD crashed. Solved.
Thanks a lot.
This question needs an answer!
Become an EE member today
7 DAY FREE TRIALMembers can start a 7-Day Free trial then enjoy unlimited access to the platform.
View membership options
or
Learn why we charge membership fees
We get it - no one likes a content blocker. Take one extra minute and find out why we block content.
that could be an issue if you are testing on a non dedicated server or if you simply set the size limit too high.
note that other than the buffer pool, many maria options will eat up memory. some globally, some for each connected client. the former will allocate ram on startup, the latter will try to allocate when a new client connects.
have a look at the error log and possibly also watch the operating system's log for OOM kills, and run a top or equivalent while testing.