Windows enable core dump

Enabling in an Option File

Core dumps are enabled by default on Windows, so this step can be skipped on Windows in those versions. See MDEV-18439 for more information.

In order to enable core dumps, you need to set the core_file system variable either on the command-line or in a relevant server option group in an option file. For example:

[mariadb]
...
core_file

You can check your current value by executing:

my_print_defaults --mariadbd

core_file is a system variable. Its value can be checked at runtime by executing the following:

SHOW GLOBAL VARIABLES LIKE 'core_file';

Core Files on Linux

There are some additional details related to using core files on Linux.

Disabling Core File Size Restrictions on Linux

On some systems there is a limit on the sizes of core files that can be dumped. You can check the system’s current system-wide limit by executing the following:

ulimit -c

You can check the current limit of the mariadbd process specifically by executing the following:

sudo cat /proc/$(pidof mariadbd)/limits | grep "core file"

If you need to change the core size limit, the method you use depends on how you start mariadbd. See the sections below for more details.

The resource limits for the mariadbd process are printed to the error log when the mariadbd process crashes. That way, users can confirm whether the process may have been allowed to dump a core file. See MDEV-15051 for more information.

Running mariadbd Using mysqld_safe

If you are starting MariaDB by running mysqld_safe, then configuring the following in the [mysqld_safe] option group in an option file should allow for unlimited sized core files:

[mysqld_safe]
...
core_file_size=unlimited

You can check your current values by executing:

my_print_defaults mysqld_safe

See mysqld_safe: Configuring the Core File Size for more details.

Note: If you are using mysqld_safe and running mariadbd as the root user, then no
core file is created on some systems. The solution is to run mariadbd as another user.

Running mariadbd Manually

If you are starting mariadbd manually or in a custom script, then you can allow for unlimited sized core files by executing the following in the same shell or script in which mariadbd is executed:

ulimit -c unlimited

Running mariadbd Using systemd

If you are starting mariadbd using systemd, then you may need to customize the MariaDB service to allow for unlimited size core files. For example, you could execute the following:

Using sudo systemctl edit mariadb.service add the contents:

[Service]

LimitCORE=infinity

See systemd: Configuring the Core File Size for more details.

Running MariaDB Containers

To get a core dump in a mariadb container requires setting the path on Linux to not include a sysctl kernel.core_pattern beginning with a pipe to an executable that doesn’t exist in the container. Setting to a straight core is recommended.

Also see Container with Debug Symbols.

Changing the System-Wide Limit

If you want to change the system-wide limit to allow for unlimited size core files for for the mysql user account, then you can do so by adding the following lines to a file in /etc/security/limits.d/. For example:

sudo tee /etc/security/limits.d/mariadb_core.conf <<EOF
mysql soft core unlimited
mysql hard core unlimited
EOF

The system would have to be restarted for this change to take effect.

See Configuring Linux for MariaDB: Configuring the Core File Size for more details.

Setting the Path on Linux

If you are using Linux, then it can be helpful to change a few settings to alter where the core files is written and what file name is used. This is done by setting the kernel.core_pattern and kernel.core_uses_pid attributes. You can check the current values by executing the following:

sysctl kernel.core_pattern
sysctl kernel.core_uses_pid

If you are using mysql-test-run and want to have the core as part of the test result, the optimal
setting is probably the following (store cores in the current directory as core.number-of-process-id):

sudo sysctl kernel.core_pattern=core.%p kernel.core_uses_pid=0

If you are using a production system, you probably want to have the core files in a specific directory,
not in the data directory. They place to store cores can be temporarily altered using the sysctl utility, but it is often more common to alter them via the /proc file system. See the following example:

sudo mkdir /tmp/corefiles
sudo chmod 777 /tmp/corefiles
sudo sysctl kernel.core_pattern=/tmp/corefiles/core
sudo sysctl kernel.core_uses_pid=1

The above commands will tell the system to put core files in /tmp/corefiles, and it also tells the system to put the process ID in the file name.

If you want to make these changes permanent, then you can add the following to a file in /etc/sysctl.conf.d/. For example:

sudo tee /etc/sysctl.d/mariadb_core.conf <<EOF
kernel.core_pattern=/tmp/corefiles/core
kernel.core_uses_pid=1
EOF

Note: if you are using containers, the pid is always going to be 1, so this may not be a useful setting. Appending an identifier like %t to the kernel.core_pattern will generate more unique files.

The value of kernel.core_pattern is printed to the error log when the mariadbd process crashes. That way, users can determine where the process may have dumped a core file. See MDEV-15051 for more information.

Note: Ensure that you have enough free disk space in the path pointed to by kernel.core_pattern.

Core dump management can be automated using systemd, which then centrally manages all core dump files and provides information about detected core dumps and access to collected core files using the coredumpctl utility.

This is enabled per default on Red Hat Enterprise Linux 8 and CentOS 8, and maybe other contemporary Linux distribution releases by now, too. It can be easily checked for by looking at the kernel.core_pattern setting. If it looks like this systemd-coredump is enabled:

# sysctl kernel.core_pattern
kernel.core_pattern = |/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e

On other distributions like Ubuntu (at least up to 21.10) it is not enabled by default, but can be set up manually.

To see all recent core dumps on the system you can then simply run

# coredumpctl list

Or you can check for MariaDB Server core dumps specifically with:

# coredumpctl list /usr/sbin/mariadbd 

If an actual core file got stored you’ll see present in the COREFILE column of the output, you can then extract the core file with:

# coredumpctl dump -o mariadbd.core ...PID...

using the process id number from the PID column, or when you just want to retrieve the latest MariaDB Server related entry:

# coredumpctl dump -o mariadb.core /usr/sbin/mariadbd

Starting with systemd 248 it is also possible to invoke the gdb debugger directly using the new --debugger-arguments=... option, e.g. making the extraction of all thread backtraces from the most recent MariaDB server crash a one liner without even having to extract the core dump file first (requires gdb to be installed):

# coredumpctl debug --debugger-arguments="-batch -ex 'thread apply all bt full'" /usr/sbin/mariadbd

So far none of the long term support Linux distribution releases have a new enough systemd version for this, the (as of this writing) still upcoming Ubuntu 22.04 «Jammy Jellyfish» will probably the first to support this.

Core Dumps and setuid on Linux

Since mariadbd executes setuid, you may have to set fs.suid_dumpable=2 to allow core dumps on Linux. You can check the current fs.suid_dumpable value by using the sysctl utility. For example:

sysctl fs.suid_dumpable

You can temporarily set it to 2 by using the sysctl utility. For example:

sudo sysctl -w fs.suid_dumpable=2

Or you can temporarily set it to 2 by writing to the /proc file system. For example:

sudo echo 2 > /proc/sys/fs/suid_dumpable

If you want to permanently set it to 2 then you can add the following to a file in /etc/sysctl.conf.d/:

sudo tee /etc/sysctl.d/mariadb_fs_suid_dumpable.conf <<EOF
fs.suid_dumpable=2
EOF

Note: If you don’t want to change fs.suid_dumpable, then another solution is to start mariadbd directly as the mysql user, so that the setuid call is not needed.

Forcing a Core File on Linux

You can get a core dump from a running server with:

sudo gcore -o filename $(pidof mariadbd)

This will store a core dump in filename.pid where pid is the process ID of mariadbd.
mariadbd will continue to be running and will not be affected by gcore.

Another method is to force a core file for mariadbd by sending the process the sigabrt signal, which has the signal code 6. This is very useful to get the state of the unresponsive mariadbd process. However, this will cause mariadbd to crash, and crash recovery will be run on restart.

You can send the signal with the kill command. For example:

sudo kill -6 $(pidof mariadbd)

As an alternative to $(pidof mariadbd), you can find the process ID either by using the ps utility or by checking the file defined by the pid_file system variable.

Core Files on Windows

Core dumps are enabled by default on Windows. See MDEV-18439 for more information.

There are some additional details related to using core files on Windows.

Minidump Files on Windows

On Windows, the core file is created as a minidump file.

For details on how to configure and read the minidump file, see How to read the small memory dump file that is created by Windows if a crash occurs.

Core Files on Kubernetes

See the IBM Core Dump Handler project.

Core Files and Address Sanitizer (ASAN)

If your mariadbd binary is built with Address Sanitizer (ASAN) then it will not be able to generate a core file.

What’s Included in Core Files

Core files usually contain a dump of all memory in the process’s full address space. This means that if a server has some large buffers configured (such as a large InnoDB buffer pool), then the server’s core files can get very large.

Some large buffers have been excluded from core files on some systems as a way to reduce the size.

The following buffers are excluded:

• InnoDB buffer pool
• InnoDB log buffer
• InnoDB Redo log buffer (fixed 2M)
• Query cache

The buffers are only excluded on Linux when using kernel version 3.4 and above and when using a non-debug build of mariadbd. Some Linux kernel versions have a bug which would cause the following warning to be printed to the log:

Sep 25 10:41:19 srv1 mysqld: 2018-09-25 10:41:19 0 [Warning] InnoDB: Failed to set memory to DODUMP: Invalid argument ptr 0x2aaac3400000 size 33554432

In those cases, the core dump may exclude some additional data. If that is not a concern, then the warning can be ignored. The problem can be fixed by upgrading to a Linux kernel version in which the bug is fixed.

See Also

• How to Produce a Full Stack Trace for mariadbd
• HowTo: Debug Crashed Linux Application Core Files Like A Pro
• A Nice Feature in MariaDB 10.3: no InnoDB Buffer Pool in Core Dumps
• Getting MySQL Core file on Linux

Photo by David Libeert

I have examined the HotSpot error files, hs_err, generated when the Java Virtual Machine (JVM) crashes (see here). A common “error” is the failure “to write core dump”.

«Failed to write coredump.»

The failure to write a core dump is however not the error causing the JVM to crash, it is merely the JVM informing you that no core dump was written.

The core dump, or “Minidump” on Windows, is a snapshot of the crashed process. If you want an analogy, you can imagine the JVM being a car driving around, and suddenly crashing into whatever, like another car, a brick wall, or a badly behaving driver. In order to give the insurance company something to look at and try to find out what happened, the JVM can take a photo of the crash site.

Different operating systems have different ways of creating core dumps. Windows has the MiniDumpWriteDump function in the dbghelp.dll DLL, it is invoked by the JVM to trigger a core dump. On the POSIX OS family (i.e. Unix, Linux & macOS), the abort() function call will make the OS generate a core dump.

The JVM argument to enable core dump writing, -XX:+CreateCoredumpOnCrash, is enabled by default on JDK 9 and above. If you are still on JDK 8, the argument is instead named -XX:+CreateMinidumpOnCrash, and this only makes sense on Windows – on other operating systems the argument is simply ignored.

Failed to write core dump. Minidumps are not enabled by default on client versions of Windows

The “client” in the “Minidumps are not enabled by default on client versions of Windows” message refers to running on a Windows client as opposed to a Windows server; not to be confused with the old HotSpot Client/Server VM distinction. On a Windows client you have to tell the JVM to write the minidump. This has nothing to do with Windows per se, so you don’t need to change any settings in Windows (there are suggestions out there, that involves enabling settings for when Windows crashes, but this doesn’t apply here). The check to see if we’re executing on a Windows client version, is explicitly made in the JVM in order to not have minidump files (which can be considerably large) fill up Windows client user’s disk space. On a Windows server however the minidumps are enabled by default.

How to enable core dumps/minidumps

JDK 9 and above

In order to enable core dumps on JDK 9 and above for any OS, you simply add the -XX:+CreateCoredumpOnCrash argument to the Java launcher.

java -jar crasher.jar -XX:+CreateCoredumpOnCrash

If you instead want to disable core dumps, you replace the plus sign with a minus sign: -XX:-CreateCoredumpOnCrash.

JDK 8 on Windows

On a client version of Windows running JDK 8, you add the -XX:+CreateMinidumpOnCrash argument to create a minidump when the JVM crashes.

java -jar crasher.jar -XX:+CreateMinidumpOnCrash

If you instead want to disable core dumps on a Windows server, you replace the plus sign with a minus: -XX:-CreateMinidumpOnCrash.

There is no JVM argument to disable core dumps in JDK 8 running on POSIX operating systems. Instead you must disable core dumps at the OS level.

Unix, Linux, and macOS core dumps

On POSIX operating systems, core dumps can be disabled on the OS level. In such case, the JVM will report that “Core dumps have been disabled”.

Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again

To control the core dump file size, you use the ulimit command, and the -c argument. You set it to 0 to disable core dumps, and to unlimited to enable them. If you run ulimit -c unlimited, you will enable core dumps for all users and all programs. So, first enable core dumps at the OS level, then run the Java program. And remember that for JDK 9 and above, you need to add the -XX:+CreateCoredumpOnCrash.

ulimit -c unlimited
java -jar crasher.jar -XX:+CreateCoredumpOnCrash

Summary of arguments

When summarized in a table one can see that in JDK 9 and above, the handling of core dumps/minidumps is more consistent. The consistency was added to JDK 9 by JDK-8074354, “Make CreateMinidumpOnCrash a new name and available on all platforms”. The two tables below summarizes the arguments and OS requirements to enable and disable core dump writing respectively. Pay attention to the plus and minus sign.

Enabling core dumps

In this table bold entries mark the setting needed to override the default behavior (i.e. only on Windows client do you need to explicitly enable core dumps).

Disabling core dumps

The second table shows how to disable core dumps. Entries in bold show where the default needs to be overridden.

What can be found in a core dump that can’t be found in the hs_err file?

The core dump is an excellent addition to the hs_err text file. To examine it, you will need a native debugger.

We can already see quite a lot of things by just looking at the generated hs_err file.

# Problematic frame:
# C  [dump.dll+0x1006]
#
# Core dump will be written. Default location: D:\dumpster\hs_err_pid29804.mdmp
[...]

Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [dump.dll+0x1006]

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j  jaokim.dumpster.Divider.native_div_call(I)I+0
j  jaokim.dumpster.Divider.do_div(I)I+2
j  jaokim.dumpster.Dumpster.do_loops(I)V+16
j  jaokim.dumpster.Dumpster.doTestcase(Ljava/lang/Integer;)V+5
j  jaokim.dumpster.Dumpster.main([Ljava/lang/String;)V+58
v  ~StubRoutines::call_stub

siginfo: EXCEPTION_INT_DIVIDE_BY_ZERO (0xc0000094)

We can see that the problematic frame where the crash happened is in native code, in dump.dll, and that a core dump was written. The native method was called by the jaokim.dumpster.Divider.native_div_call Java method. We can also see what seems to be some sort of divide by zero exception (EXCEPTION_INT_DIVIDE_BY_ZERO). If we however want to get more details from the native code in dump.dll, the hs_err file won’t help us more than this!

To get those missing details, we have to use a native debugger (ex. WinDbg on Windows) and the generated core dump. Below is a WinDbg screenshot pointing directly at the crashing line!

To find the exact source line, the native debugger needs to have the debugging symbols. For Windows, these are in the .pdb file generated when the dump.dll is compiled. (You can put the .pdb file in the same directory as the coredump in order for WinDbg to find it.) On POSIX OS, the shared object (dump.so) needs to be compiled with debug symbols turned on.

Summary

When the JVM crashes, it can write a core dump that contains details of the crashed process. If core dump writing is disabled, the JVM will tell you it failed to write a core dump – this has however nothing to do with the actual reason the JVM crashed. The reason for the crash can, in most cases, be found in the hs_err file, and more intricate details can be found in the core dump.

Creating core dumps to help analyze VirtualBox crashes

A core dump is a technical term for a file containing detailed information about the state of an application at a given time, in a form which developers can make use of. They are particularly helpful for tracking down the reasons for application crashes, which is why many systems can be told to create one automatically when an application crash occurs.

How to create a core dump on Linux

VirtualBox is (as of version 2.0.0) a system application which can be started from a user account without special privileges — a so-called «setuid» application. As core dumps are usually not allowed for these sensitive applications you have to explicitly allow them after starting your computer but before the crash which you want a core dump from. This can be done by typing the following command at the command line (your user account must be allowed to use the «sudo» command):

$ echo -n 1 | sudo tee /proc/sys/fs/suid_dumpable

To create a core dump on a SUSE system

​https://www.suse.com/support/kb/doc/?id=3054866

The procedure for actually creating a core dump is similar on most Linux systems, but on Ubuntu it is slightly different due to the Ubuntu crash reporting tool (Apport).

To create a core dump on an Ubuntu system

create a file called «.config/apport/settings» in your home directory with the contents

[main]
unpackaged=true

and run VirtualBox. Any crashes of VirtualBox after you have created the file will cause a crash report to be created in /var/crash with a name like «_usr_lib_virtualbox_VirtualBox.1000.crash». Do not accept the offer to «Send an error report to help fix this problem». You can extract the core file using the «apport-unpack» tool, or if you are unsure you can just send us the full crash report file. To stop crash files being created for non-system applications delete the file that you created again.

To create a core dump on non-Ubuntu systems

start VirtualBox from a command line (e.g. xterm):

$ ulimit -c unlimited
$ echo -n 1 | sudo tee /proc/sys/fs/suid_dumpable
$ VirtualBox

or if possible start the virtual machine directly:

$ ulimit -c unlimited
$ echo -n 1 | sudo tee /proc/sys/kernel/core_uses_pid
$ /usr/lib/virtualbox/VirtualBox -startvm VM_NAME

Ensure that no startup script (~/.bashrc, ~/.bash_profile, ~/.profile) contains an instruction like ulimit -c 0 as the limit cannot be increased once it was set to zero.

When VirtualBox or one of its processes crashes, a file core.<pid> is created in the current directory. Be aware that core dumps can be very huge. Please compress the file before submitting it to a bug report. Or better don’t attach the file to a report. Note that this core dump can contain a memory dump of your guest which can include sensitive information. Send it to alexander _dot_ eichner _at_ oracle _dot_ com if the compressed file is smaller than 5MB. Contact me directly otherwise so we can look for another solution to get the core dump to us.

If several core files are created, you can check which process created them using the command

$ file core.<pid>

to be sure of the right one to send.

How to create dumps on Mac OS X

To create a core dump on Mac OS X, start VirtualBox from a command line:

$ ulimit -c unlimited
$ VirtualBox

or better start the VM directly:

$ ulimit -c unlimited
$ /Applications/VirtualBox.app/Contents/MacOS/VirtualBox -startvm VM_NAME

Ensure that no startup script (~/.bashrc, ~/.bash_profile, ~/.profile) contains an instruction like ulimit -c 0 as the limit cannot be increased once it was set to zero.

Alternatively (for Finder & launchpad)

$ launchctl limit core unlimited

or for permanent effect the same could be specified in $HOME/.launchd.conf or /etc/launchd.conf.

The core files can be found in the /cores folder. That directory must be writable by the program that’s dumping core. By default, it is only writable by root and by users in the admin group. If you need normal users to be able to dump a core, you can make the directory world writable with the command:

$ sudo chmod o+w /cores

How to create dumps on Solaris

To enable core dumps on Solaris, run the following command as root

# coreadm -g /var/cores/core.%f.%p -i core.%f.%p \
  -e global -e process -e global-setid -e proc-setid -e log

The cores would end up in /var/cores/ folder. Global dumps would end up in /var/crash/<hostname>/

System core dumps need to be enabled via dumpadm. The important thing is to have «Savecore enabled» to «yes» (use dumpadm -y). The configuration should look something like this:

# dumpadm
      Dump content: kernel pages
       Dump device: /dev/zvol/dsk/rpool/dump (dedicated)
Savecore directory: /var/crash/myhostname
  Savecore enabled: yes
   Save compressed: on

Now to obtain a core dump for the VM process, run:

VBoxManage startvm <vmname>

Forcing VirtualBox to terminate with a core dump

Sometimes it is required to force a VirtualBox process to terminate, for example, a VM hangs for some unknown reason. On Linux, this can be done as follows:

$ ulimit -c unlimited
$ sudo echo -n 1 > /proc/sys/fs/suid_dumpable
$ /usr/lib/virtualbox/VirtualBox -startvm VM_NAME &
$ pidof VirtualBox
7145
$ kill -4 7145

As an alternative to kill you can do

$ pidof VirtualBox
7145
$ gcore 7145

On Mac OS X:

$ ulimit -c unlimited
$ /Applications/VirtualBox.app/Contents/MacOS/VirtualBox -startvm VM_NAME &
$ ps aux|grep VirtualBox
... 7145 ... VirtualBox ...
$ kill -4 7145

On Solaris:

# ulimit -c unlimited
# /opt/VirtualBox/amd64/bin/VirtualBox -startvm VM_NAME &
# ps -ef|grep VirtualBox
... 7145 ... VirtualBox ...
# kill -4 7145

You can find result core file according location specified in coreadm

# coreadm

Passing the signal number 4 (SIGILL) is essential! The same applies to the alternative frontends VBoxHeadless and VBoxSDL.

Minidumps on Windows

On Windows so-called Mini Dumps are available for application crashes and/or investigating purposes. To enable creating such Mini Dumps for a specific application, Windows offers the following tools:

• For Windows XP, Vista, 7, 8(.1) and Windows 10: Windows Error Reporting (WER)
• For older Windows versions such as Windows NT 4 or Windows 2000: Dr. Watson. VirtualBox is not supported on these platforms anymore.

If you got a BSOD (Bluescreen) instead, meaning that your PC was unable recover and needed to be restarted as soon as you used VirtualBox, you need to get a kernel dump instead.

Enabling Mini Dumps using Windows Error Reporting (WER)

• Open up the regedit.exe as Administrator and create the key

  HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Windows Error Reporting\LocalDumps
  

  if it does not exist yet.

• Create a registry REG_DWORD value named DumpType with a value of 2 (means full dump).
• Create a registry REG_SZ value named DumpFolder containing a path to a folder where to store the Mini Dumps, e.g. C:\Windows\Temp.

The next time VirtualBox crashes you’ll find a newly created Mini Dump in the folder you specified above. Before sending those Mini Dumps to us though, please first compress them with ZIP (e.g. 7zip) or RAR to reduce the overall file size. This also saves you time uploading this file!

For more detailed information visit ​this page.

Using the Process Explorer

The process explorer (Windows Sysinternals) allows to create a minidump or a full application dump using the context menu. Of course this helps only if the process didn’t terminate due to a crash.

Creating a kernel dump when a Bluescreen (BSOD) occurred

Windows automatically should create a kernel dump for you if a Bluescreen occurred. However, you might want to enable creating a more verbose kernel dump which then would help us a bit better.

To do so, go (as Administrator) to the System failure settings by

Start -> Control Panel -> System -> Advanced system settings (left side) -> Tab "Advanced" -> "Startup and Recovery"

On this page in the System failure area you can change the «Write debugging information» to Kernel memory dump.
The Dump file path shows you where this file is being created then — the default location is C:\Windows\MEMORY.DMP.

Please also be aware that each time a Bluescreen occurred the file mentioned above will be overwritten! This means that you should copy this file to a save place to not lose it if you want to provide us the right information. Also, other applications on your PC may produce Bluescreens, so only send us the kernel dump which got created when directly using VirtualBox (and your PC showed a Blueescreen).

Note: Before sending those kernel dumps to us though, please first compress them with ZIP (e.g. 7zip) or RAR to reduce the overall file size. This also saves you time uploading this file!

Privacy information: Also be aware that the above kernel dumps could contain unrelated sensitive and private information about you and your system, e.g. stored passwords in memory. Unfortunately this is unavoidable in those situations, as a kernel dump essentially is an unmodified and unfiltered part of your computer’s RAM (main memory).