Mysql segfaulting when used from the shell

Question

I have a Raspberry Pi server with Raspbian OS:

Kernel: Linux 4.9.35+ #1014 Fri Jun 30 14:34:49 BST 2017 armv6l GNU/Linux
Description:    Raspbian GNU/Linux 8.0 (jessie)
Release:        8.0
Codename:       jessie

Today I noticed that attempts to use mysql end in a segmentation fault.

user@host~ $ mysql -u root -p
Enter password:
Segmentation fault

This happens for both the wrong and the right password. Or even if I made up a username. Actually, it turns out than even running the mysql command without any arguments has the same effect.

The Mysql server can still be accessed via Python (pymysql) and Perl. I have scripts that write and read various DBs, they are all working without problems.

Shell scripts that use the mysql command, they all fail. For example:

/home/user/example.sh: line 2: 27974 Segmentation fault      /usr/bin/mysql -u dbuser -p$dbpass dbname --execute="select * from example;"

The segmentation faults started appearing today and I cannot figure what is causing them now. The server has not been booted in a few weeks. It has been more than a week since it was last updated.

I can't find any errors that might look like relevant to this situation from the Mysql logs or syslog.

I have tried:

• Restarting Mysql
• Upgrading the system and rebooting
• Checking disk on reboot, no errors found

As these procedures didn't help, I tried using gdb as suggested here:

Running application ends with "Segmentation Fault"

This is what I get when debugging the command mysql without any parameters:

gdb mysql run run Starting program: /usr/bin/mysql

gdb mysql
run
Starting program: /usr/bin/mysql
Program received signal SIGSEGV, Segmentation fault.
elf_dynamic_do_Rel (skip_ifunc=<optimized out>, lazy=0, nrelative=<optimized out>, relsize=<optimized out>,
    reladdr=<optimized out>, map=0xb6fff968) at do-rel.h:112
112     do-rel.h: No such file or directory.

I wonder what could I do to fix this problem? (Other than making a bug report about this.)

Answer 1

You most likely have corrupted binaries/and or a corrupted filesystem/SD card.

The SD cards are not meant for heavy I/O use and degrade over time; Raspberry(es) are also known to corrupt data in SD cards when turning off occasionally due to characteristics of their design (electronics is not my area, not able to get into details).

You might very well have a corruption on the mysql binary or associated libraries. (actually having a gdb failure at do-rel.h suggests the latter).

I would reinstall the mysql client and associated libraries, as a command similar to this one (your mileage may vary):

sudo apt-get install --reinstall default-mysql-client default-mysql-client-core

I would use this command to see what package is giving you the mysql binary and would reinstall it:

dpkg -S /usr/bin/mysql

Then I would also see what libraries mysql is using, if that does not fix the problem:

ldd /usr/bin/mysql
    linux-vdso.so.1 (0x00007ffc8903c000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f5989c75000)
    libreadline.so.5 => /lib/x86_64-linux-gnu/libreadline.so.5 (0x00007f5989a33000)
    libncurses.so.5 => /lib/x86_64-linux-gnu/libncurses.so.5 (0x00007f5989810000)
    libtinfo.so.5 => /lib/x86_64-linux-gnu/libtinfo.so.5 (0x00007f59895e6000)
    libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f59893cc000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f59891c8000)
    libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f5988e46000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f5988b42000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f59887a3000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f598a4bc000)
    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f598858c000)

You might have as last resort to reinstall each package supporting each one of this libraries until your have your error corrected. Some of them are: libaio1, libjemalloc1, libreadline5. There are more.

sudo apt-get install -reinstall libaio1 libjemalloc1 libreadline5

Nevertheless, there are no guarantees you do not have other bits of your filesystem corrupted. I would backup the DB and reinstall the OS/MySQL from scratch.

The good news is that you mentioning other ways of accessing the DB are working well, it means the corruption is mostly related to the mysql binary client.

Nonetheless, I would probably reevaluate running Linux from an SD card card in the future, especially if using MySQL.

PS. As @cas points well, "if you have dlocate or debsums installed, you can run dlocate --md5check PKGNAME or debsums PKGNAME to verify the package's installed files against its md5sum file"

See Raspberry: booting from a USB pen instead of an SD card