How to solve a timeout problem in MariadB

I had my 10.2.15-MariaDB server crashed after a select count(*) from a 18.6 Gb table.

 It's not a HeidiSQL error because it also happens if i do it using MariaDB's command prompt.

After this crash, i can't do a select count(*) over 60 seconds that i get the same error again.

It doesn't seem like a configuration issue as this weren't happening before the first crash.

I used to run several hours querys without a problem.

My interactive_timeout variable is set to 28800:

"Variable_name";"Value" "connect_timeout";"36000" "delayed_insert_timeout";"300" "innodb_flush_log_at_timeout";"1" "innodb_lock_wait_timeout";"50" "innodb_rollback_on_timeout";"OFF" "interactive_timeout";"28800" "lock_wait_timeout";"86400" "net_read_timeout";"30" "net_write_timeout";"60" "slave_net_timeout";"60" "thread_pool_idle_timeout";"60" "wait_timeout";"28800"

I still have 62 Gb free in my hard drive (SSD) and my PC has 32 Gb. of RAM memory.

Thanks!
Fabio Avila CastroSenior analistAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

skullnobrainsCommented:
if your buffer pool is bigger than the available RAMswap space, maria will crash.
that could be an issue if you are testing on a non dedicated server or if you simply set the size limit too high.

note that other than the buffer pool, many maria options will eat up memory. some globally, some for each connected client. the former will allocate ram on startup, the latter will try to allocate when a new client connects.

have a look at the error log and possibly also watch the operating system's log for OOM kills, and run a top or equivalent while testing.
Fabio Avila CastroSenior analistAuthor Commented:
I´m not a MariadB or database expert.
So, I have some difficulties to figure out what´s going wrong.

My error log from MariadB

Server version: 10.2.15-MariaDB
key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=1
max_threads=65537
thread_count=7
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 136060 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
mysqld.exe!my_sigabrt_handler()[my_thr_init.c:487]
mysqld.exe!raise()[signal.cpp:516]
mysqld.exe!abort()[abort.cpp:71]
mysqld.exe!os_file_handle_error_cond_exit()[os0file.cc:5227]
mysqld.exe!os_aio_windows_handler()[os0file.cc:6574]
mysqld.exe!os_aio_handler()[os0file.cc:5701]
mysqld.exe!fil_aio_wait()[fil0fil.cc:5350]
mysqld.exe!io_handler_thread()[srv0start.cc:338]
KERNEL32.DLL!BaseThreadInitThunk()
ntdll.dll!RtlUserThreadStart()
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
2019-03-01 22:36:44 12560 [Note] InnoDB: Mutexes and rw_locks use Windows interlocked functions
2019-03-01 22:36:44 12560 [Note] InnoDB: Uses event mutexes
2019-03-01 22:36:44 12560 [Note] InnoDB: Compressed tables use zlib 1.2.3
2019-03-01 22:36:44 12560 [Note] InnoDB: Number of pools: 1
2019-03-01 22:36:44 12560 [Note] InnoDB: Using SSE2 crc32 instructions
2019-03-01 22:36:44 12560 [Note] InnoDB: Initializing buffer pool, total size = 8G, instances = 8, chunk size = 128M
2019-03-01 22:36:44 12560 [Note] InnoDB: Completed initialization of buffer pool
2019-03-01 22:36:45 12560 [Note] InnoDB: Highest supported file format is Barracuda.
2019-03-01 22:36:45 12560 [Note] InnoDB: Starting crash recovery from checkpoint LSN=1281846889734
2019-03-01 22:36:45 12560 [Note] InnoDB: 128 out of 128 rollback segments are active.
2019-03-01 22:36:45 12560 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2019-03-01 22:36:45 12560 [Note] InnoDB: Creating shared tablespace for temporary tables
2019-03-01 22:36:45 12560 [Note] InnoDB: Setting file '.\ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2019-03-01 22:36:45 12560 [Note] InnoDB: File '.\ibtmp1' size is now 12 MB.
2019-03-01 22:36:45 12560 [Note] InnoDB: Waiting for purge to start
2019-03-01 22:36:45 12560 [Note] InnoDB: 5.7.22 started; log sequence number 1281846889743
2019-03-01 22:36:45 5216 [Note] InnoDB: Loading buffer pool(s) from C:\Program Files\MariaDB 10.2\data\ib_buffer_pool
2019-03-01 22:36:45 12560 [Note] Plugin 'FEEDBACK' is disabled.
2019-03-01 22:36:45 12560 [Note] Reading of all Master_info entries succeded
2019-03-01 22:36:45 12560 [Note] Added new Master_info '' to hash table
2019-03-01 22:36:45 12560 [Note] C:\Program Files\MariaDB 10.2\bin\mysqld.exe: ready for connections.
Version: '10.2.15-MariaDB'  socket: ''  port: 0  mariadb.org binary distribution
2019-03-01 22:37:15 5216 [Note] InnoDB: Buffer pool(s) load completed at 190301 22:37:15
2019-03-02  0:29:19 3528 [ERROR] InnoDB: Operating system error number 23 in a file operation.
2019-03-02  0:29:19 3528 [Note] InnoDB: Some operating system error numbers are described at https://mariadb.com/kb/en/library/operating-system-error-codes/
2019-03-02  0:29:19 3528 [ERROR] InnoDB: File .\teste\dados_2013.ibd: 'Windows aio' returned OS error 223. Cannot continue operation
190302  0:29:19 [ERROR] mysqld got exception 0x80000003 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.

Server version: 10.2.15-MariaDB
key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=1
max_threads=65537
thread_count=7
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 136060 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
2019-03-02  0:29:19 12956 [ERROR] InnoDB: Operating system error number 23 in a file operation.
2019-03-02  0:29:19 12956 [Note] InnoDB: Some operating system error numbers are described at https://mariadb.com/kb/en/library/operating-system-error-codes/
2019-03-02  0:29:19 12956 [ERROR] InnoDB: File .\teste\dados_2013.ibd: 'Windows aio' returned OS error 223. Cannot continue operation
2019-03-02  0:48:42 9648 [Note] InnoDB: Mutexes and rw_locks use Windows interlocked functions
2019-03-02  0:48:42 9648 [Note] InnoDB: Uses event mutexes
2019-03-02  0:48:42 9648 [Note] InnoDB: Compressed tables use zlib 1.2.3
2019-03-02  0:48:42 9648 [Note] InnoDB: Number of pools: 1
2019-03-02  0:48:42 9648 [Note] InnoDB: Using SSE2 crc32 instructions
2019-03-02  0:48:42 9648 [Note] InnoDB: Initializing buffer pool, total size = 8G, instances = 8, chunk size = 128M
2019-03-02  0:48:42 9648 [Note] InnoDB: Completed initialization of buffer pool
2019-03-02  0:48:42 9648 [Note] InnoDB: Highest supported file format is Barracuda.
2019-03-02  0:48:42 9648 [Note] InnoDB: Starting crash recovery from checkpoint LSN=1281846889771
2019-03-02  0:48:43 9648 [Note] InnoDB: 128 out of 128 rollback segments are active.
2019-03-02  0:48:43 9648 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2019-03-02  0:48:43 9648 [Note] InnoDB: Creating shared tablespace for temporary tables
2019-03-02  0:48:43 9648 [Note] InnoDB: Setting file '.\ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2019-03-02  0:48:43 9648 [Note] InnoDB: File '.\ibtmp1' size is now 12 MB.
2019-03-02  0:48:43 9648 [Note] InnoDB: Waiting for purge to start
2019-03-02  0:48:43 9648 [Note] InnoDB: 5.7.22 started; log sequence number 1281846889780
2019-03-02  0:48:43 12924 [Note] InnoDB: Loading buffer pool(s) from C:\Program Files\MariaDB 10.2\data\ib_buffer_pool
2019-03-02  0:48:43 9648 [Note] Plugin 'FEEDBACK' is disabled.
2019-03-02  0:48:43 9648 [Note] Reading of all Master_info entries succeded
2019-03-02  0:48:43 9648 [Note] Added new Master_info '' to hash table
2019-03-02  0:48:43 9648 [Note] C:\Program Files\MariaDB 10.2\bin\mysqld.exe: ready for connections.
Version: '10.2.15-MariaDB'  socket: ''  port: 0  mariadb.org binary distribution
2019-03-02  0:49:10 12924 [Note] InnoDB: Buffer pool(s) load completed at 190302  0:49:10

Open in new window

skullnobrainsCommented:
forget the memory issue. you may still run into some with your settings but that was not the cause of the crash

the interesting information is here

2019-03-02  0:29:19 3528 InnoDB: Operating system error number 23 in a file operation.
2019-03-02  0:29:19 3528 [Note] InnoDB: Some operating system error numbers are described at https://mariadb.com/kb/en/library/operating-system-error-codes/
2019-03-02  0:29:19 3528 InnoDB: File .\teste\dados_2013.ibd: 'Windows aio' returned OS error 223. Cannot continue operation
190302  0:29:19 mysqld got exception 0x80000003 ;

i would assume you are using file_per_table and dados_2013 is a table in teste database which would be corrupt.

can you confirm that table is the one involved ?
CEOs need to know what they should worry about

Nearly every week during the past few years has featured a headline about the latest data breach, malware attack, ransomware demand, or unrecoverable corporate data loss. Those stories are frequently followed by news that the CEOs at those companies were forced to resign.

skullnobrainsCommented:
i'm unsure but i guess you are hitting

ERROR_FILE_TOO_LARGE

    223 (0xDF)

    The file size exceeds the limit allowed and cannot be saved.

described here https://docs.microsoft.com/en-us/windows/desktop/debug/system-error-codes--0-499-

which seems strange because ntfs should allow for HUGE sizes.
can you check how big the file is and if it can be opened in whatever editor will allow to open very big files ?
Fabio Avila CastroSenior analistAuthor Commented:
Yes, the table dados_2013 was involved in the first crash and after that Maria crash in other simple querys with another tables. Sometimes not
But dados_2013 was involved in the first crash after a simple : select count (*) from dados_2013.
Fabio Avila CastroSenior analistAuthor Commented:
The table dados_2013 has 20 Gb. I could open it in UltraEdit, but the free trial has expired. So, it´s a test that I can not run now.

PS1: After crash I have to restart MySQL process in the task manager to reconnect with MariaDb.
PS2: I can drop dados_2013 from dB. I have a backup.
skullnobrainsCommented:
whatever the running query, you can check whether the filename that appears in the logs is always the same. this should help determine between a problem with this specific file or you're hitting filesystem corruption / disk problems.

you have not answered regarding the filesize. you may or may not be reaching a well-known limit.

is your system 32 bits ? or possibly your mariadb version ?

ultraedit has tons of free alternatives. i'm pretty sure you can figure that out. you can also try to zip the file which will force the archieving program to read the whole file and complain in case there is corruption. obviously try while mariadb is not running.

regards
Fabio Avila CastroSenior analistAuthor Commented:
The DB files are corrupted due to disk problem.
HD crashed. Solved.
Thanks a lot.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Databases

From novice to tech pro — start learning today.